General Asylum: Thanks... by PhilNYC

General Asylum: Thanks... by PhilNYC

General audio topics that don't fit into specific categories.

FAQ / News / Events

Thanks...

66.43.89.26

In Reply to: A/B testing methodology question posted by PhilNYC on August 20, 2006 at 07:46:50:

Thanks for all the feedback on this test. As noted, I certainly wasn't trying to accomplish anything on a true scientific level, but I thought it was a fun casual experiment and was curious to hear what those who are scientific-testing-inclined thought. Particularly with this group of audiophiles whom I know very well (the NY Audio Rave on Audiocircle.com), I wasn't too worried about "peer pressure" influencing the results, although I do acknowledge the possibility that there could have been some kind of subconscious effect. The reason why this particular test stood out to me compared to others was how, with eyes closed, a significant number of folks did raise their hands at exactly the "right" moment...seemed like there must have been something to trigger the response.

This post is made possible by the generous support of people like you and our sponsors:

Parts Connexion

Topic - A/B testing methodology question - PhilNYC 07:46:50 08/20/06 ( 76)

Re: A/B testing methodology question - tianguis 09:57:54 08/26/06 ( 0)
Perfectly valid test to me. - hukkfinn 08:13:36 08/24/06 ( 0)
The trick is: - Presto 13:48:24 08/21/06 ( 0)
Do you have a polarity switch on your preamp? - Dave Pogue 10:01:12 08/21/06 ( 4)
- Shhhh... you might attract the attention of that polarity psycho! :) [nt] - Steve Cortez 16:41:12 08/21/06 ( 2)
  - Excuse me! That's Polarity Psychoâ„¢,ok? We MUST stay abreast of such things...N/T - musetap 18:02:43 08/21/06 ( 0)
  - Lord, I hope not. (nt) - Dave Pogue 16:51:21 08/21/06 ( 0)
- Re: Do you have a polarity switch on your preamp? - PhilNYC 10:15:25 08/21/06 ( 0)
A slight change in volume (SPL) will be audible ... but so what? - Richard BassNut Greene 08:58:32 08/21/06 ( 5)
- Re: A slight change in volume (SPL) will be audible ... but so what? - PhilNYC 09:28:41 08/21/06 ( 4)
  - You didn't mention if the tweak caused a small change in volume (which could be audible) nt - Richard BassNut Greene 09:30:35 08/22/06 ( 3)
    - Re: You didn't mention if the tweak caused a small change in volume (which could be audible) nt - PhilNYC 12:01:08 08/22/06 ( 2)
      - A small change in level will often be mistaken for a change in quality - Analog Scott 12:05:11 08/22/06 ( 1)
        
        Re: A small change in level will often be mistaken for a change in quality - PhilNYC 13:39:25 08/22/06 ( 0)
Thanks... - PhilNYC 08/21/06 08:43:42 08/21/06 ( 0)
You will NEVER EVER find a test that will satisfy your antagonists.... - clarkjohnsen 08:31:28 08/21/06 ( 9)
- Re: You will NEVER EVER find a test that will satisfy your antagonists.... - Inmate51 09:11:02 08/21/06 ( 6)
  - "Clark is clearly being sarcastic." No, the word is "realistic". Those people will never be happy... - clarkjohnsen 09:13:23 08/21/06 ( 5)
    - I don't think you are being realistic at all. - Analog Scott 15:48:47 08/21/06 ( 4)
      - Re: I don't think you are being realistic at all. - PhilNYC 09:06:22 08/22/06 ( 1)
        
        Re: I don't think you are being realistic at all. - Analog Scott 10:13:45 08/22/06 ( 0)
      - "His protocols were painfully poor for a meaningful blind test." See what I mean? - clarkjohnsen 08:43:30 08/22/06 ( 1)
        
        Re: "His protocols were painfully poor for a meaningful blind test." See what I mean? - Analog Scott 10:21:53 08/22/06 ( 0)
- I have no antagonists... ;-) - PhilNYC 08:57:34 08/21/06 ( 1)
  - Aw, c'mon... nt - clarkjohnsen 09:16:58 08/21/06 ( 0)
it's a pretty good test but it has a big problem - tunenut 18:51:53 08/20/06 ( 2)
- Re: it's a pretty good test but it has a big problem - PhilNYC 19:51:34 08/20/06 ( 1)
  - Re: it's a pretty good test but it has a big problem - tunenut 21:02:50 08/20/06 ( 0)
Sounds like a fun session :) I'm awfully curious what the tweak was, I've tried something similar... - Steve Cortez 17:53:22 08/20/06 ( 0)
Re: A/B testing methodology question - Inmate51 12:58:21 08/20/06 ( 0)
Make lots of tests - Caymus 11:22:00 08/20/06 ( 0)
It was a great test, Phil; way to go. Obviously the fans of DBT don't accept it... - jeffreybehr 10:05:18 08/20/06 ( 3)
- Re: It was a great test, Phil; way to go. Obviously the fans of DBT don't accept it... - Bob Wortman 18:15:03 08/20/06 ( 2)
  - I believe most of us would label the former as being OPEN-minded, not close-minded. - jeffreybehr 19:51:27 08/20/06 ( 1)
    - Arrogance is Them, alright. nt - clarkjohnsen 08:24:17 08/21/06 ( 0)
probably not because.. - Tom Tripp 09:22:12 08/20/06 ( 2)
- Re: probably not because.. - PhilNYC 09:44:57 08/20/06 ( 1)
  - Re: probably not because.. - Tom Tripp 09:53:40 08/20/06 ( 0)
a better way, i think - pingong 09:14:20 08/20/06 ( 0)
Re: A/B testing methodology question - Pat D 09:09:16 08/20/06 ( 8)
- Re: A/B testing methodology question - PhilNYC 09:33:03 08/20/06 ( 7)
  - Re: A/B testing methodology question - Analog Scott 11:42:53 08/20/06 ( 6)
    - Uhh...Scott...read it all. NO ONE knew WHAT he was going to do... - jeffreybehr 12:17:06 08/20/06 ( 5)
      - "Everybody was certainly aware that I was going to be implementing a tweak" - Analog Scott 12:40:46 08/20/06 ( 4)
        
        Not a thing. I'm responding to your statement... - jeffreybehr 14:37:12 08/20/06 ( 3)
        
        They did know what he was going to do. - Analog Scott 15:02:18 08/20/06 ( 2)
        
        Re: They did know what he was going to do. - PhilNYC 15:44:53 08/20/06 ( 1)
        
        That might have added some uncertainty. - Analog Scott 16:07:51 08/20/06 ( 0)
Re: A/B testing methodology question - Norm 08:43:09 08/20/06 ( 2)
- Re: A/B testing methodology question - PhilNYC 08:55:12 08/20/06 ( 1)
  - Be happy! nt - Norm 08:56:16 08/20/06 ( 0)
Not valid - Analog Scott 08:29:05 08/20/06 ( 6)
- Itâ€™s still better than sighted listening tests - Caymus 11:46:08 08/20/06 ( 1)
  - I disagree. One could get unfounded confidence - Analog Scott 13:23:54 08/20/06 ( 0)
- Great response, Scott...NOT. Do you think those who raised their hands DID NOT hear a change? - jeffreybehr 09:53:51 08/20/06 ( 3)
  - Re: no isolation from social pressure - mls-stl 12:39:10 08/20/06 ( 0)
  - Who knows? - Slider 11:25:31 08/20/06 ( 0)
  - i think there is no way to know, because you have too many variables. - Analog Scott 11:13:12 08/20/06 ( 0)
You would get the same results if you did nothing. (nt) - Garth 08:24:44 08/20/06 ( 15)
- What Garth is saying is: - Frihed89 10:29:08 08/20/06 ( 0)
- Re: You would get the same results if you did nothing. (nt) - PhilNYC 08:51:27 08/20/06 ( 13)
  - Re: You would get the same results if you did nothing. (nt) - Analog Scott 11:44:31 08/20/06 ( 4)
    - I thought... - PhilNYC 12:00:23 08/20/06 ( 3)
      - Re: I thought... - Analog Scott 12:54:03 08/20/06 ( 0)
      - Phil, try to pay no attention to Scott. Your testing methodology is perfectly valid... - jeffreybehr 12:23:27 08/20/06 ( 1)
        
        What a bunch of balony. - Analog Scott 12:57:18 08/20/06 ( 0)
  - If you did nothing and none raised their hands you would have some validation. - Norm 08:55:14 08/20/06 ( 7)
    - Re: If you did nothing and none raised their hands you would have some validation. - PhilNYC 08:57:39 08/20/06 ( 6)
      - Re: Test administrator needs to be blind as well... - mls-stl 09:31:49 08/20/06 ( 5)
        
        Re: Test administrator needs to be blind as well... - PhilNYC 09:49:36 08/20/06 ( 3)
        
        Actually few experiments are done double blind - Norm 10:40:17 08/20/06 ( 2)
        
        Re: same thing stated another way - mls-stl 11:29:55 08/20/06 ( 1)
        
        I was merely pointing out that double blind testing is relatively uncommon - Norm 11:45:58 08/21/06 ( 0)
        
        Good point about not generalizing to what others might hear. nt - Norm 09:40:25 08/20/06 ( 0)
Re: A/B testing methodology question - Todd Krieger 08:21:21 08/20/06 ( 1)
- Re: A/B testing methodology question - PhilNYC 08:50:24 08/20/06 ( 0)