In Reply to: A/B testing methodology question posted by PhilNYC on August 20, 2006 at 07:46:50:
I'd be more concerned with what I could hear during a single blind comparison (SBT) using a good test methodology, not what others claimed they could hear (we really don't know for sure if they DID hear anything using your test methodology).A SBT can usually be done in one hour, with one assistant, at no cost --- that so many audiophiles avoid these easy-to-do learning experiences is a shame.
That so many people raised their hands in your "test" at the same time suggests the tweak altered the volume (meaningless) ... or frequency response (meaningful).
I hope "installing" the tweak did not cause any break in the music, or the noise of a switch, because both would be clues to the listeners that something had been changed.
Based on controlled listening tests, audiophiles will typically report hearing differences 50% to 75% of the time when comparing a component with itself, so this bias needs to be eliminated if you want useful test results. (With these tests there is no difference in A-B volumes possible because only ONE component is used -- but small SPL differences are almost certain when comparing TWO components).
There are five types of A-B component differences -- your test methodology did not eliminate the meaningless differences:
(1) Slight change in SPL = audible but meaningless
(2) Imagined differences = meaninglessThe types of differences we want to know about:
(3) Differences heard by some listeners with some recordings on some systems (if only one person "hears differences" in a blind test, that is a good reason to suspect lucky guessing on his score card, and do the test again)(4) Differences heard by most listeners with most recordings on most systems
(Even (3) and (4) could be meaningless to you ... if YOU couldn't hear the differences using your own system!)
.
.
.
Richard BassNut Greene
My Stereo is MUCH BETTER than Your Stereo
This post is made possible by the generous support of people like you and our sponsors:
Follow Ups
- A slight change in volume (SPL) will be audible ... but so what? - Richard BassNut Greene 08/21/0608:58:32 08/21/06 (5)
- Re: A slight change in volume (SPL) will be audible ... but so what? - PhilNYC 09:28:41 08/21/06 (4)
- You didn't mention if the tweak caused a small change in volume (which could be audible) nt - Richard BassNut Greene 09:30:35 08/22/06 (3)
- Re: You didn't mention if the tweak caused a small change in volume (which could be audible) nt - PhilNYC 12:01:08 08/22/06 (2)
- A small change in level will often be mistaken for a change in quality - Analog Scott 12:05:11 08/22/06 (1)
- Re: A small change in level will often be mistaken for a change in quality - PhilNYC 13:39:25 08/22/06 (0)