What you can learn from a split test (and my BIG mistake!)*

[dots and spots Jeff]

Inspired by a thread a few days ago, I bit the bullet and set up my first split test, using Google'e Website Optimizer.


I was worried that my homepage was too busy & cluttered, so I created a second test page by removing many of the images to see which converted better.

Here are some screen grabs of the two pages:

http://www.ukbusinessforums.co.uk/forums/picture.php?albumid=714&pictureid=3201 The original page (you can view it 'live' here (http://www.dotsandspots.biz/index.html))

http://www.ukbusinessforums.co.uk/forums/picture.php?albumid=714&pictureid=3202 The new, uncluttered, page (you can view it 'live' here (http://www.dotsandspots.biz/indexB.html))

Before I spill the beans and let you know what I found out, what it has taught me about split testing and the BIG mistake I made I invite anyone to suggest which version (if any) performed better.

Jeff

*(One thing I have learnt is the power of a headline that grabs the attention - I've tried to write one here!)

[MrsPWN]

Well I much prefer the 1st one, I am your target audience (female and love STUFF lol)

The second one is too bare and I am not sure I would bother going any further. If you think it is too cluttered have you looked at having the products on a white background? Although I think the cluttered look is chic ;)

Interested to see your mistake....

[Big Pete]

Well i guess the 2nd performs better .. but 1st is better for keeping a customer onsite longer .. so maybe a compromise is needed..;)

[Steve Cool]

I can't explain why, but if i landed on your site, the first would make me carry on and on the second I would hit the back button.
Suppose it's like looking through the window of a high street shop with no money to spend on stock.

Steve

[gemd89]

I would be more likely to spend time on the 1st site

Gemma

[MrsPWN]

Based on the reactions of others I would say the mistake is taking the 2nd option to the extreme and not just making subtle changes....

[Raw Rob]

Don't think the second one will work as well. Three of the four image links take you away from your site (to your blog, twitter and facebook). You want to keep people on your site so they buy, not give them easy ways to leave!

Rob

[fiona davies]

Got to be the second one. You are talking to me in a very subtle way with the line...'gift's you will want to send and give'. You are making me want to go into the site and spend!!! :) You are calling me to action and have told me that my action is to send and give your products (and not SPEND money) It's very subtle.

keep us posted!

[SillyJokes]

Is the mistake that you got the link to your shop wrong on the second page?

Your call to action SHOP doesn't work.

[SteveGibson]

I'd go for the first one.

Steve

PS The approx 50:50 spread of opinions on here just goes to show why testing is important.

[dots and spots Jeff]

Is the mistake that you got the link to your shop wrong on the second page?

Your call to action SHOP doesn't work.

Oops - no! That's a second big mistake (one that I've now corrected.) Thanks.

[SillyJokes]

Oops - no! That's a second big mistake (one that I've now corrected.) Thanks.

My work here is done. Night night all.

[dots and spots Jeff]

Wow - I'm amazed and grateful for all your responses.

As Steve says, just going by your opinions (which are all very welcome and valid) would leave me none the wiser, hence the need for accurate data.

I'll leave it a little longer to see if anyone else has an opinion, but I promise I'll reveal the data and the lessons I've learnt (oh yes - and tell you my mistake!)

Thanks again to all,

Jeff

[seedstotal]

the changes are major, hard to say what was ur mistake, i cant wait
1st version for me

[JamieM]

Oops - no! That's a second big mistake (one that I've now corrected.) Thanks.

Does that not mean the results are void?

[Scott-CopyandDesign]

Does that not mean the results are void?

I'd say yes.

[SteveGibson]

Does that not mean the results are void?

Good question.

Steve

[Mediamonitor]

1st one is my pref - really interested to see what you learnt.

[UKSBD]

New For 2010

[Martin-S]

Having the View Cart button more visible must be a good thing. 2nd page also has a visible call to action on my monitor without scrolling which would help too.

[termsandconditions]

My punt is Option 2 in view of the clearer call to action.

[I, Brian]

Both look like they are targeted at different people and therefore ad groups - ie, homepage for generic keywords, and the second page as a targeted landing page for valentine's day.

I think the potential *big* mistake is if you're trying to send *everybody* to the homepage, instead of into very targeted landing areas of your site.

2c.

[admagic]

Come on , let us out of our misery.

I have to say neither, because neither tells me what you are selling, why I should bother reading, why I should buy it from you...so in either case, I would click away

[BrightIdeas]

I also think the first one performed better. Second is a bit bare and doesn't have that cosy chic feel!

[awebapart.com]

I'm not going to comment on which version I think was more successful, because you haven't told us enough information about the test and your definition of success, and there might be other issues which invalidate your test anyway:

1. You haven't told us what your success criteria is, is it selling valentines cards or selling more of any of your products, is it getting to a certain point in the checkout process, or is it just getting people to sign up to your newsletter.

2. For a seasonal market like your market, where certain products are more popular than others at certain times of the year, e.g. valentines cards, having a split test in this way, one focusing on valentines cards, the other not, might invalidate the test

3. You have not said how people will be arriving at your 2 pages, e.g. mailshot, adwords, etc, what pre-site introductory text was displayed, and how many unique visitors you were getting a day. Are you really getting enough visitors in a short time to get a meaningful statistic?

4. If you are not narrowing down the test to visitors from a particular source, e.g. adwords, then your test might be tainted by other visitors, e.g. people from this site landing on your pages

5. Your results might also be tainted by having 2 home pages, the domain . biz home page and the domain . biz / index.html home page

6. Your results might also be tainted by the fact that visitors to your indexb.htm home page will still see your other home page if they click on the home link

Other general mistakes I would comment on include:

1. Having a go at onsite parallel split testing, which I would consider an advanced technique (especially parallel home page split testing which is the more complex to implement correctly than say non-home page landing page split testing), before you have sorted out the basics on your site. Trying to run before you can walk. Focus on the basics first. One basic mistake that might be seriously affecting sales is the trust issue, lack of company information, bad characters in your terms and conditions page when viewed in Firefox, and probably the worst issue of them all, a personal btinternet email address rather than company name when visitors eventually get to your paypal paying area. These basic things can be easily remedied and they should be addressed before you even think of doing anything advanced.

2. Having a go at your own onsite parallel split testing rather than making use of the results of other company's split testing. This is especially relevant for small companies who do not have the resources to effectively perform their own onsite parallel split testing properly, or do not have the visitor traffic to gain any valid statistical result. Put simply, copy the big successful boys. Learn from what the successful stores do. What do they do? Well it is not rocket science to suggest that successful online stores in seasonal markets (cards, fashion, etc) emphasise different relevant seasonal things on their home page, but not necessarily taking away the mention of their other products too. Take a look at sites like Moonpig, Clinton Cards, Next, and many more sites so you can see common themes emerging.

3. Is your split testing going to mess up other marketing channels on your site? For instance your test B page is now indexed by google and this could effect your search engine rankings and traffic. If you don't want test pages indexed by google then take the necessary steps to prevent this.

This might seem critical, but I hope it is passing on some useful information too. And finally, I can say without a doubt, that whichever test home page you have, it is certainly better than the current hallmark.co.uk home page, which is still trying to flog Christmas cards as its main theme if you visit the site using a browser without Flash installed!

[awebapart.com]

New For 2010
I'm not sure whether you picked up on this earlier feedback Jeff, but there are multiple links under the text "New For 2010" on your home page (test A), and if you click on the links under "For" or "2010" you get a page not found error (going to cards.html.html).

This link error could also mess up the results of your comparison test as the text, link and error is only present in one of the tests.

This error could also have been affecting your sales well before you started the split test, since the error was present before the test, it is there in Google's Jan 19th cache of your site.

[dots and spots Jeff]

First of all, Thank you to all of you who have taken the time to read this thread and add your thoughts.

I should start my 'report' by reminding you all that I am a complete newb to split testing - I certainly don't believe I can teach the old hands anything, but maybe one or two other 'beginners' can learn/be encouraged by what I tried.

Inspired by another thread - and lots of great help and encouragement of others on the forum - I thought I'd have a go at doing a split test. I read, watched a few videos and read a little more, but I believe to really learn and understand you need to do, so I did, and my little experiment, however flawed taught me so much more than just reading and watching.

First, I was amazed at how easy it was to set up a split test with Google website optimizer, so if you haven't done a test before, don't let the technical challenge of setting up the test be an obstacle - it needn't be.

If anything, it was perhaps a little too easy to set up the test and, in my eagerness to experiment with testing, I didn't think enough about the test - what I wanted to test, why, how and what would constitute a success. Next time I think I'd sit down with a piece of paper and jot down a sentence on what, why & how before I even went near the computer.

Resistance to testing: I don't really have a design bone in my body - my OH is the designer behind dots and spots. She designed (I built) the original page and was horrified to hear that I was even considering changing it. She liked it (the original), couldn't see the need to test or change anything. I suspect this attitude may be prevalent in lots of business's, from the very small to the very big.

Discipline: once I'd set the test running I kept wanting to go and check the data. Once I started checking the data, the temptation to pull one of the pages was strong.

Before I started, I thought the test would only run on when someone clicked through from a Google search - I didn't really understand the mechanics of the whole process. I now realise that, because of the code you add, either page will be served up, regardless of where the visitor arrived from.

I didn't add Google analytics to my test page, so I don't have the normal analytics data on that page.

And so which page performed best? Well, with the caveat of my BIG mistake (I'm saving that for the end!) throughout the test, the second page out performed the first page, although shortly before stopping the test, they began to converge, with the first page converting 54.8% of the time, the second page converting 60% of the time - a conversion being when someone either clicked through to our Valentine Page or our shop page - so I suppose neither really outdid the other.

So what was my BIG mistake - well once the test was up and running I resalised I'd only defined a conversion as going to our shop or valentines page, but there are a number of other valid actions the user could have taken.

So someone landing on the original home page and choosing to go to one of the pages other than the Valentines shop page would count as a fail in terms of the test but if they went to, say, our cards in reality this isn't a fail, so I realised I wasn't really testing like for like and whilst you might be able to say more people visited our Valentine page from the second version, I can't really tell from the test results which site made the users do something 'useful' - which may, or may not, have been visiting the Valentine page.

Anyway, I fear I'm rambling now and maybe not making sense.

Theory is all well and good but you need practical experience. I may not have gained a great insight into consumer preferences and the physcology of buying through this experiment, but I have learnt a lot about split testing and how to do it (although I know I've a whole lot more left to learn!)

But if you've never done any testing like this before, don't be afraid, have a go, I know I will!

Thanks again to all for your input & help,

Jeff

[SteveGibson]

I'd only defined a conversion as going to our shop or valentines page

That's why the second page won: the first one had options you arbitrarily defined as "non conversions", the second one cut the choices down to (pretty much) only things you were counting as "success".

Steve

PS You should learn from this and do another test.

[dots and spots Jeff]

That's why the second page won: the first one had options you arbitrarily defined as "non conversions", the second one cut the choices down to (pretty much) only things you were counting as "success".

Steve

PS You should learn from this and do another test.

This is what dawned on me 24 hours into the test - oh well, we learn from our mistakes and I won't let it put me off doing another test.

A big personal thanks for all your help and inspiration in getting me 'testing' - I hope one or two others (or more!) who've read this thread also take up testing.

Jeff

[admagic]

This is what dawned on me 24 hours into the test - oh well, we learn from our mistakes and I won't let it put me off doing another test.

A big personal thanks for all your help and inspiration in getting me 'testing' - I hope one or two others (or more!) who've read this thread also take up testing.

Jeff

Great stuff. When you start testing you never look back.

One very important point which will sound facile, but is very true.

The easiest way to get somebody to take the action is to ASK them to do it, and give them a reason why!!

Your results will jump the moment you give them a reason to do what you want them to do....the better the reason , the more will do it..

[1weekSEO]

Hi Jeff

I wished I had read this earlier but I was tearing my hair out yesterday with my own GA.

I'd go for the first page every time, maybe it's because I'm female, but I have been to your site before and I remember you posting last year about a Facebook page for your business.

I agree with the person who said the 2nd page was a shop window with no stock - a bit like a handbag shop with just one handbag! I would bounce.

Please let us know when you do your second test and thank you for sharing this experience with us.

Nina

[SteveGibson]

This is what dawned on me 24 hours into the test - oh well, we learn from our mistakes and I won't let it put me off doing another test.

Great. As long as you stay committed to testing, you're going to succeed. It's just a matter of time.

It's worth pointing that the first test I set up didn't even work! (didn't rotate the versions - I should have read the instructions properly!) So you got further than I did.

A big personal thanks for all your help and inspiration in getting me 'testing' - I hope one or two others (or more!) who've read this thread also take up testing.

You're welcome. I hope your testing - the good and the bad - inspires others to give it a go. Please keep sharing your progress.

Steve