PDA

View Full Version : Sample range to judge impact of changes on conversions?


DuaneJackson
20th June 2009, 14:40
I'm always trying to improve the conversion rate of visitor to sign up.

After making a big change, how much data do you need to accurately judge whether a change was worthwhile?

Do you wait until you have a whole months worth of data? Or if you saw a reasonably large change in conversions within a week would you accept that as being indicative of it working (or not)?

I've made a change and seen a big (negative) difference in conversions within a week. I'm wondering should I change back now as it's obviously detrimental, or hold off for a few weeks in case there's some external factor involved?

directmarketingadvice
20th June 2009, 14:58
I've made a change and seen a big (negative) difference in conversions within a week. I'm wondering should I change back now as it's obviously detrimental, or hold off for a few weeks in case there's some external factor involved?

That's why you have to split-test.

Sequential testing - "website used to look like this, now it looks like that" - results contain so much noise, it's really just guessing.

(as you've just found out)

Steve