In Reply to: A/B testing methodology question posted by PhilNYC on August 20, 2006 at 07:46:50:
As described below, you always employed the unnamed tweak.If you want to do a better test, do this.
Start the music either with the tweak or without the tweak. Ask people to decide whether the tweak is IN or OUT. Do something random each time to decide whether it will be IN or OUT (like flip a coin).
Do this at least 10 times and see how many people have recognized the tweak being IN or OUT correctly.
This procedure eliminates the highly predictable nature of your experiment.
Or if you feel that the immediate switching is important, wait exactly some time interval and either switch the tweak IN or do nothing. Same idea as above. Do at least 10 trials and count up the results.
This post is made possible by the generous support of people like you and our sponsors:
Follow Ups
- it's a pretty good test but it has a big problem - tunenut 08/20/0618:51:53 08/20/06 (2)
- Re: it's a pretty good test but it has a big problem - PhilNYC 19:51:34 08/20/06 (1)
- Re: it's a pretty good test but it has a big problem - tunenut 21:02:50 08/20/06 (0)