What you can learn from a split test (and my BIG mistake!)*

SillyJokes said:

Is the mistake that you got the link to your shop wrong on the second page?

Your call to action SHOP doesn't work.

Click to expand...

Oops - no! That's a second big mistake (one that I've now corrected.) Thanks.

Wow - I'm amazed and grateful for all your responses.

As Steve says, just going by your opinions (which are all very welcome and valid) would leave me none the wiser, hence the need for accurate data.

I'll leave it a little longer to see if anyone else has an opinion, but I promise I'll reveal the data and the lessons I've learnt (oh yes - and tell you my mistake!)

Thanks again to all,

Jeff

First of all, Thank you to all of you who have taken the time to read this thread and add your thoughts.

I should start my 'report' by reminding you all that I am a complete newb to split testing - I certainly don't believe I can teach the old hands anything, but maybe one or two other 'beginners' can learn/be encouraged by what I tried.

Inspired by another thread - and lots of great help and encouragement of others on the forum - I thought I'd have a go at doing a split test. I read, watched a few videos and read a little more, but I believe to really learn and understand you need to do, so I did, and my little experiment, however flawed taught me so much more than just reading and watching.

First, I was amazed at how easy it was to set up a split test with Google website optimizer, so if you haven't done a test before, don't let the technical challenge of setting up the test be an obstacle - it needn't be.

If anything, it was perhaps a little too easy to set up the test and, in my eagerness to experiment with testing, I didn't think enough about the test - what I wanted to test, why, how and what would constitute a success. Next time I think I'd sit down with a piece of paper and jot down a sentence on what, why & how before I even went near the computer.

Resistance to testing: I don't really have a design bone in my body - my OH is the designer behind dots and spots. She designed (I built) the original page and was horrified to hear that I was even considering changing it. She liked it (the original), couldn't see the need to test or change anything. I suspect this attitude may be prevalent in lots of business's, from the very small to the very big.

Discipline: once I'd set the test running I kept wanting to go and check the data. Once I started checking the data, the temptation to pull one of the pages was strong.

Before I started, I thought the test would only run on when someone clicked through from a Google search - I didn't really understand the mechanics of the whole process. I now realise that, because of the code you add, either page will be served up, regardless of where the visitor arrived from.

I didn't add Google analytics to my test page, so I don't have the normal analytics data on that page.

And so which page performed best? Well, with the caveat of my BIG mistake (I'm saving that for the end!) throughout the test, the second page out performed the first page, although shortly before stopping the test, they began to converge, with the first page converting 54.8% of the time, the second page converting 60% of the time - a conversion being when someone either clicked through to our Valentine Page or our shop page - so I suppose neither really outdid the other.

So what was my BIG mistake - well once the test was up and running I resalised I'd only defined a conversion as going to our shop or valentines page, but there are a number of other valid actions the user could have taken.

So someone landing on the original home page and choosing to go to one of the pages other than the Valentines shop page would count as a fail in terms of the test but if they went to, say, our cards in reality this isn't a fail, so I realised I wasn't really testing like for like and whilst you might be able to say more people visited our Valentine page from the second version, I can't really tell from the test results which site made the users do something 'useful' - which may, or may not, have been visiting the Valentine page.

Anyway, I fear I'm rambling now and maybe not making sense.

Theory is all well and good but you need practical experience. I may not have gained a great insight into consumer preferences and the physcology of buying through this experiment, but I have learnt a lot about split testing and how to do it (although I know I've a whole lot more left to learn!)

But if you've never done any testing like this before, don't be afraid, have a go, I know I will!

Thanks again to all for your input & help,

Jeff

SteveGibson said:

That's why the second page won: the first one had options you arbitrarily defined as "non conversions", the second one cut the choices down to (pretty much) only things you were counting as "success".

Steve

PS You should learn from this and do another test.

Click to expand...

This is what dawned on me 24 hours into the test - oh well, we learn from our mistakes and I won't let it put me off doing another test.

A big personal thanks for all your help and inspiration in getting me 'testing' - I hope one or two others (or more!) who've read this thread also take up testing.

Jeff