General Asylum: Re: A/B testing methodology question by Todd Krieger

Topic - A/B testing methodology question - PhilNYC 07:46:50 08/20/06 ( 76)

Re: A/B testing methodology question - tianguis 09:57:54 08/26/06 ( 0)
Perfectly valid test to me. - hukkfinn 08:13:36 08/24/06 ( 0)
The trick is: - Presto 13:48:24 08/21/06 ( 0)
Do you have a polarity switch on your preamp? - Dave Pogue 10:01:12 08/21/06 ( 4)
- Shhhh... you might attract the attention of that polarity psycho! :) [nt] - Steve Cortez 16:41:12 08/21/06 ( 2)
  - Excuse me! That's Polarity Psychoâ„¢,ok? We MUST stay abreast of such things...N/T - musetap 18:02:43 08/21/06 ( 0)
  - Lord, I hope not. (nt) - Dave Pogue 16:51:21 08/21/06 ( 0)
- Re: Do you have a polarity switch on your preamp? - PhilNYC 10:15:25 08/21/06 ( 0)
A slight change in volume (SPL) will be audible ... but so what? - Richard BassNut Greene 08:58:32 08/21/06 ( 5)
- Re: A slight change in volume (SPL) will be audible ... but so what? - PhilNYC 09:28:41 08/21/06 ( 4)
  - You didn't mention if the tweak caused a small change in volume (which could be audible) nt - Richard BassNut Greene 09:30:35 08/22/06 ( 3)
    - Re: You didn't mention if the tweak caused a small change in volume (which could be audible) nt - PhilNYC 12:01:08 08/22/06 ( 2)
      - A small change in level will often be mistaken for a change in quality - Analog Scott 12:05:11 08/22/06 ( 1)
        
        Re: A small change in level will often be mistaken for a change in quality - PhilNYC 13:39:25 08/22/06 ( 0)
Thanks... - PhilNYC 08:43:42 08/21/06 ( 0)
You will NEVER EVER find a test that will satisfy your antagonists.... - clarkjohnsen 08:31:28 08/21/06 ( 9)
- Re: You will NEVER EVER find a test that will satisfy your antagonists.... - Inmate51 09:11:02 08/21/06 ( 6)
  - "Clark is clearly being sarcastic." No, the word is "realistic". Those people will never be happy... - clarkjohnsen 09:13:23 08/21/06 ( 5)
    - I don't think you are being realistic at all. - Analog Scott 15:48:47 08/21/06 ( 4)
      - Re: I don't think you are being realistic at all. - PhilNYC 09:06:22 08/22/06 ( 1)
        
        Re: I don't think you are being realistic at all. - Analog Scott 10:13:45 08/22/06 ( 0)
      - "His protocols were painfully poor for a meaningful blind test." See what I mean? - clarkjohnsen 08:43:30 08/22/06 ( 1)
        
        Re: "His protocols were painfully poor for a meaningful blind test." See what I mean? - Analog Scott 10:21:53 08/22/06 ( 0)
- I have no antagonists... ;-) - PhilNYC 08:57:34 08/21/06 ( 1)
  - Aw, c'mon... nt - clarkjohnsen 09:16:58 08/21/06 ( 0)
it's a pretty good test but it has a big problem - tunenut 18:51:53 08/20/06 ( 2)
- Re: it's a pretty good test but it has a big problem - PhilNYC 19:51:34 08/20/06 ( 1)
  - Re: it's a pretty good test but it has a big problem - tunenut 21:02:50 08/20/06 ( 0)
Sounds like a fun session :) I'm awfully curious what the tweak was, I've tried something similar... - Steve Cortez 17:53:22 08/20/06 ( 0)
Re: A/B testing methodology question - Inmate51 12:58:21 08/20/06 ( 0)
Make lots of tests - Caymus 11:22:00 08/20/06 ( 0)
It was a great test, Phil; way to go. Obviously the fans of DBT don't accept it... - jeffreybehr 10:05:18 08/20/06 ( 3)
- Re: It was a great test, Phil; way to go. Obviously the fans of DBT don't accept it... - Bob Wortman 18:15:03 08/20/06 ( 2)
  - I believe most of us would label the former as being OPEN-minded, not close-minded. - jeffreybehr 19:51:27 08/20/06 ( 1)
    - Arrogance is Them, alright. nt - clarkjohnsen 08:24:17 08/21/06 ( 0)
probably not because.. - Tom Tripp 09:22:12 08/20/06 ( 2)
- Re: probably not because.. - PhilNYC 09:44:57 08/20/06 ( 1)
  - Re: probably not because.. - Tom Tripp 09:53:40 08/20/06 ( 0)
a better way, i think - pingong 09:14:20 08/20/06 ( 0)
Re: A/B testing methodology question - Pat D 09:09:16 08/20/06 ( 8)
- Re: A/B testing methodology question - PhilNYC 09:33:03 08/20/06 ( 7)
  - Re: A/B testing methodology question - Analog Scott 11:42:53 08/20/06 ( 6)
    - Uhh...Scott...read it all. NO ONE knew WHAT he was going to do... - jeffreybehr 12:17:06 08/20/06 ( 5)
      - "Everybody was certainly aware that I was going to be implementing a tweak" - Analog Scott 12:40:46 08/20/06 ( 4)
        
        Not a thing. I'm responding to your statement... - jeffreybehr 14:37:12 08/20/06 ( 3)
        
        They did know what he was going to do. - Analog Scott 15:02:18 08/20/06 ( 2)
        
        Re: They did know what he was going to do. - PhilNYC 15:44:53 08/20/06 ( 1)
        
        That might have added some uncertainty. - Analog Scott 16:07:51 08/20/06 ( 0)
Re: A/B testing methodology question - Norm 08:43:09 08/20/06 ( 2)
- Re: A/B testing methodology question - PhilNYC 08:55:12 08/20/06 ( 1)
  - Be happy! nt - Norm 08:56:16 08/20/06 ( 0)
Not valid - Analog Scott 08:29:05 08/20/06 ( 6)
- Itâ€™s still better than sighted listening tests - Caymus 11:46:08 08/20/06 ( 1)
  - I disagree. One could get unfounded confidence - Analog Scott 13:23:54 08/20/06 ( 0)
- Great response, Scott...NOT. Do you think those who raised their hands DID NOT hear a change? - jeffreybehr 09:53:51 08/20/06 ( 3)
  - Re: no isolation from social pressure - mls-stl 12:39:10 08/20/06 ( 0)
  - Who knows? - Slider 11:25:31 08/20/06 ( 0)
  - i think there is no way to know, because you have too many variables. - Analog Scott 11:13:12 08/20/06 ( 0)
You would get the same results if you did nothing. (nt) - Garth 08:24:44 08/20/06 ( 15)
- What Garth is saying is: - Frihed89 10:29:08 08/20/06 ( 0)
- Re: You would get the same results if you did nothing. (nt) - PhilNYC 08:51:27 08/20/06 ( 13)
  - Re: You would get the same results if you did nothing. (nt) - Analog Scott 11:44:31 08/20/06 ( 4)
    - I thought... - PhilNYC 12:00:23 08/20/06 ( 3)
      - Re: I thought... - Analog Scott 12:54:03 08/20/06 ( 0)
      - Phil, try to pay no attention to Scott. Your testing methodology is perfectly valid... - jeffreybehr 12:23:27 08/20/06 ( 1)
        
        What a bunch of balony. - Analog Scott 12:57:18 08/20/06 ( 0)
  - If you did nothing and none raised their hands you would have some validation. - Norm 08:55:14 08/20/06 ( 7)
    - Re: If you did nothing and none raised their hands you would have some validation. - PhilNYC 08:57:39 08/20/06 ( 6)
      - Re: Test administrator needs to be blind as well... - mls-stl 09:31:49 08/20/06 ( 5)
        
        Re: Test administrator needs to be blind as well... - PhilNYC 09:49:36 08/20/06 ( 3)
        
        Actually few experiments are done double blind - Norm 10:40:17 08/20/06 ( 2)
        
        Re: same thing stated another way - mls-stl 11:29:55 08/20/06 ( 1)
        
        I was merely pointing out that double blind testing is relatively uncommon - Norm 11:45:58 08/21/06 ( 0)
        
        Good point about not generalizing to what others might hear. nt - Norm 09:40:25 08/20/06 ( 0)
Re: A/B testing methodology question - Todd Krieger 08/20/06 08:21:21 08/20/06 ( 1)
- Re: A/B testing methodology question - PhilNYC 08:50:24 08/20/06 ( 0)