In Reply to: A/B testing methodology question posted by PhilNYC on August 20, 2006 at 07:46:50:
A brute-force way to minimize statistical errors is to gather as many test samples as possible. To randomize the selection, us a coin-toss or dice roll. This way they won’t expect to hear something different every test (just as likely to be the same as the previous test).You’ll need to take pains to not tip off what to expect. For example when I test audiophiles for detecting cables, even if the test is a repeat of the previous setup, I’ll still unplug the cable, set it down with the others, then pick it up and re-connect it. This way the same amount of time is used, the same setup cues, etc. I start the test saying nothing and then leave the room for the duration of the test. Make sure you’re behavior is consistent.
Testing a group is significantly harder because they can easily influence each other making the results meaningless.
This post is made possible by the generous support of people like you and our sponsors:
Follow Ups
- Make lots of tests - Caymus 08/20/0611:22:00 08/20/06 (0)