Conversion optimization

[Toni Anicic]

When you are split testing your product or checkout page variations, how much time and what sample of visitors you use in order to be sure you got a representative sample?

[SteveGibson]

When you are split testing your product or checkout page variations, how much time and what sample of visitors you use in order to be sure you got a representative sample?

I usually use Website Optimizer and run it until there's at least 90% confidence.

(or that it seems to be making no difference)

Steve

[phillipsinternet]

When I worked for a Google Consultant we ran tests for at least a month or until the result went red or green. For it to go green it needs to hit 98%, and google recommend not stopping a test until confidence hits 95%+.

It usually takes a lot of data to hit this so you'll need to ensure you have enough traffic. I've seen tests at 90% drop to 60% very quickly.

I don't understand the stats fully myself but in short it is a lot easier to go from 90% to 95% than from 95% to 96% (so I've been told by my statistician mate).

Oh, and run 100% of visitors through the test.

[SteveGibson]

When I worked for a Google Consultant we ran tests for at least a month or until the result went red or green. For it to go green it needs to hit 98%, and google recommend not stopping a test until confidence hits 95%+.

When I studied statistics at Uni, the level used was 95%.

However, that was for clinical/theoretical questions that required absolute answers.

For "which is more profitable" tests, I think 90% makes more sense as the chances of a 90% confidence level reversing is very low - it'll either go up to 95% or go inconclusive.

And, if it goes inconclusive (and stays there), you've got to pick one version
at random anyway.

(the exception is for freak results based on very low data samples)

Steve

PS I've forgotten the guy's name, but one of the supposed testing experts said he uses 80%. Personally, I think this is too low, but it might make sense given the number of extra tests you can do.

[phillipsinternet]

I agree 100% on that, and also about 90% being ok if there's not enough data being sampled. One e-commerce site we've tested gets around 17 conversions a day, the tests take ages to run so we never go all the way to 98%.

One freak occurrance we had was a test that went red :( within a few days and not much data. We thought that Website Optimiser had made a mistake but it never reversed. So I guess in some circumstances you don't need large amounts of data (maybe just for losing tests? I should have done stats not mechanics :rolleyes:)

[SteveGibson]

One freak occurrance we had was a test that went red :( within a few days and not much data. We thought that Website Optimiser had made a mistake but it never reversed. So I guess in some circumstances you don't need large amounts of data (maybe just for losing tests? I should have done stats not mechanics :rolleyes:)

I had a test for a client where it quickly went to around 93% then went down to around 40% and never really budged from there.

The reason I didn't call it a winner at 90%+ was that the difference in the copy was so small, it couldn't have realistically caused the conversion rate to go up by 150-200%.

Having that sort of reality check - based on experience of past tests - is important when the sample is small.

Steve

[Toni Anicic]

I thanked everyone, these were very useful comments!