In Reply to: Re: A/B testing methodology question posted by PhilNYC on August 20, 2006 at 09:33:03:
"Everybody was certainly aware that I was going to be implementing a tweak. But their eyes were closed from the start to finish for each run, and I did not say anything while the music was playing the entire time."If everybody knew what you were going to do the test was contaminated. I suggest repeating it and randomly not apply the tweak. See what happens then. The problem with eyes closed is that people can still sense time passed, your movement, other peoples' movements etc. Like I said before. Tooooo many variables.
This post is made possible by the generous support of people like you and our sponsors:
Follow Ups
- Re: A/B testing methodology question - Analog Scott 08/20/0611:42:53 08/20/06 (6)
- Uhh...Scott...read it all. NO ONE knew WHAT he was going to do... - jeffreybehr 12:17:06 08/20/06 (5)
- "Everybody was certainly aware that I was going to be implementing a tweak" - Analog Scott 12:40:46 08/20/06 (4)
- Not a thing. I'm responding to your statement... - jeffreybehr 14:37:12 08/20/06 (3)
- They did know what he was going to do. - Analog Scott 15:02:18 08/20/06 (2)
- Re: They did know what he was going to do. - PhilNYC 15:44:53 08/20/06 (1)
- That might have added some uncertainty. - Analog Scott 16:07:51 08/20/06 (0)