Topic: Using a meet to do some blind testing? (Read 1079 times)

khaos · « **on:** April 13, 2012, 05:29:22 AM »

I wonder if a meet could be used to do some blind testing. Let's not fool anyone, the objective would not be to establish the audibility of x and y piece of gear, but rather to get a starting point of discussion. I would imagine the following setup:

- a quiet room with a CD player or a dac feeding 4 or 5 headamps simultaneously, perhaps a computer with a large and diverse playlist?
- all headamps volume matched within 0.1 dB with each a pair HD800 plugged in, the amps would be hidden to view and the headphones given a number. Volume control would either be impossible or left to the CDP/DAC if it has a preamp function.
- the HD 800 is chosen because Sennheiser seems to have achieved a remarkable level of consistency between every pair, any other consistent and detailed headphones would do.
- perhaps 3 low distortion, low impedance, low noise amps and 2 other designs.
- a rating sheet where the listener would give marks to bass/mids/treble quality, quantity in relation to each other, clarity, details... some mathematical crunching later on would be done to first normalize the marks for each person, and then establish a pattern with each amp.
- at least one control, ie. 2 different HD800s (known to sound very closely) plugged into the same amplifier model, perhaps using the twin output of an amp?

And perhaps the most important thing would be to ask the listeners to not share their impressions with other people, so that that listeners are are free of any preconceived notions when submitted to the test.

At the end of the day, some number crunching would be done to find out how similar people rate each amp according to the parameters used and how it compares to the control.

So anyone think this could be interesting, or that it could be done?

Anaxilus. · « **Reply #1 on:** April 13, 2012, 06:17:30 AM »

Of course it could be done, the question is should it or is it worth the effort. I do believe most if not all of the 'studies' presented to date are inherently flawed in one way or another. I have a few concerns:

1-What level of significance do we seek to attribute to this exercise? Are we offering definitive evidence or messing about? If the latter, do we possibly risk giving value to a largly worthless endeavor?

2-How extreme and how much effort is acceptable to meet this goal?

3-Do we have the will and manpower to meet this goal?

4-Matched gains will have to be relative to each persons desired gain at which they claim or believe to hear differences (Fletcher/Munson's). This will not be universal to all people but universal to the participant.

5-Participants should be familiar w/ the test gear used ahead of time and use the same gear for the test but done blind.

6-Participants should use only tracks they know intimately or they all become intimate w/ the same tracks they believe they can discern.

7-External pressures (social, financial, whatever) have to minimized as much as possible.

8-Samples and switching must fall within short bursts around 25-45 secs give or take.

9-Each individual test session can't last more than 3-5 minutes, give or take, otherwise listening fatigue and stress sets in and everything sounds the same.

10-We need to account for more unknowns than listed here so far. I would seek out input from those that have a lot of genuine, well established regimens for AB that don't talk out their asses.

These are just general guidelines based on my own experiences offered for consumption. I'll volunteer to take such a test. Purrin tried to sneak a 320kbps past me this afternoon. Naughty boy.

khaos · « **Reply #2 on:** April 13, 2012, 07:12:26 AM »

Of course it could be done, the question is should it or is it worth the effort. I do believe most if not all of the 'studies' presented to date are inherently flawed in one way or another. I have a few concerns:
Indeed, whether it could garner enough interest is the main point of this thread, mind you it's purely theroritically on my part, I won't fly to the US for a meet.

1-What level of significance do we seek to attribute to this exercise? Are we offering definitive evidence or messing about? If the latter, do we possibly risk giving value to a largly worthless endeavor?
I think it would be awfully presumptuous to say it offers definitive evidence, but it's not just to mess about, efforts should be undertaken to give it a minimum of rigour.

2-How extreme and how much effort is acceptable to meet this goal?3-Do we have the will and manpower to meet this goal?
Financially, it does imply a separate room in a meet, otherwise enough people to lend the gear and someone willing to crunch numbers, otherwise, this is the purpose of this thread.

4-Matched gains will have to be relative to each persons desired gain at which they claim or believe to hear differences (Fletcher/Munson's). This will not be universal to all people but universal to the participant.
Matched gain within 0.5 dB is easy, use a voltmeter to calibrate all the amps and you're done. Matched within 0.1 dB is harder, it depends on how consistent the sensitivity of the different pairs of HD800 is. One would have to measure them and tweak the potentiometer of each amp to match said sensitivity. But the HD800 are very consistent, it may not be an issue.

On the other hand, Fletcher-Munson has nothing to do with it, with a general volume control (perhaps the 32 bit VC of Foobar on PC) placed upstream to all the amps, you set the volume of all amps simultaneously, so any difference heard between the amps would not be due to volume differences.

5-Participants should be familiar w/ the test gear used ahead of time and use the same gear for the test but done blind.
Yes, that's an issue, but any blind testing relying on a large amount of participants instead of a large amounts of trials to gather statistical data is going to run into the same one. At least, the HD800 is a model that most experienced headphone audiophiles have experience with.

6-Participants should use only tracks they know intimately or they all become intimate w/ the same tracks they believe they can discern.
Allowing participants to bring their own music would mitigate the issue.

7-External pressures (social, financial, whatever) have to minimized as much as possible.
Make it anonymous?

8-Samples and switching must fall within short bursts around 25-45 secs give or take.
That's an issue only if the result is negative (no difference between amps), in this case, switching is as simple as taking off one pair of HD 8000 and putting on another pair.

9-Each individual test session can't last more than 3-5 minutes, give or take, otherwise listening fatigue and stress sets in and everything sounds the same.
Yes, I'm familiar with the issues of DBT, in this case it's not exactly a DBT, it's about how much the auditory impressions gathered when listening to a headphone amplifier differ when blind. In other words, it's about analogous to what we do everytime we go to a meet or a hifi store, except in more controlled conditions.

10-We need to account for more unknowns than listed here so far. I would seek out input from those that have a lot of genuine, well established regimens for AB that don't talk out their asses.
Indeed, it's difficult to proof against negative results due to testing conditions, that's why I don;t think it should be considered anything definitive, but I would hope that after this, more people would consider to carry out their own experimentations at home in more rigorous conditions.

These are just general guidelines based on my own experiences offered for consumption. I'll volunteer to take such a test. Purrin tried to sneak a 320kbps past me this afternoon. Naughty boy.
Did he manage to do so?

Anax modded-restored original post

raif · « **Reply #3 on:** April 13, 2012, 07:59:31 AM »

I would like to make two superlative-type statements.

1) I have never seen seen anyone pass a [fill in "blind" related procedure here] test in audio with regards to the usual subjects.(cables, 320kbps vs lossless, different properly implemented SS amps within their driving capabilities, etc.)

2) I have never seen it change their mind if they didn't want it to.

I like to think I am of the science mind with respect to audiophool tendencies yet I still have to spend about 30 minutes a day convincing myself that I don't NEED a BHSE to squeeze the last ounce of performance out of my OII. (I think I am losing the battle.)

Anaxilus. · « **Reply #4 on:** April 13, 2012, 08:02:27 AM »

Financially, it does imply a separate room in a meet, otherwise enough people to lend the gear and someone willing to crunch numbers, otherwise, this is the purpose of this thread.
More than that if we agree on sample gear familiarity. I'm including sources and amps in question so loaners/acquisition and familiarization time is a complication.

On the other hand, Fletcher-Munson has nothing to do with it, with a general volume control (perhaps the 32 bit VC of Foobar on PC) placed upstream to all the amps, you set the volume of all amps simultaneously, so any difference heard between the amps would not be due to volume differences.
Right, I was just concerned w/ variation among test subjects, not gear. If they have master control it's moot.

Make it anonymous?
And unpublicized or widely circulated, no privacy sacrifices w/ youtube vids.

That's an issue only if the result is negative (no difference between amps), in this case, switching is as simple as taking off one pair of HD 8000 and putting on another pair.
To elaborate, the track would ideally be reset if headphone switching is used and the idea was to limit or control ear/brain fatigue and psychoacoustics.

Did he manage to do so?
Someone was scolded. It wasn't me.

I think the line was something like, "are you using MP3's to try to f**k with me again?"

His chain is just too transparent.

Maxvla · « **Reply #5 on:** April 13, 2012, 08:19:15 AM »

Quote from: raif on April 13, 2012, 07:59:31 AM

I would like to make two superlative-type statements.

1) I have never seen seen anyone pass a [fill in "blind" related procedure here] test in audio with regards to the usual subjects.(cables, 320kbps vs lossless, different properly implemented SS amps within their driving capabilities, etc.)

2) I have never seen it change their mind if they didn't want it to.

I like to think I am of the science mind with respect to audiophool tendencies yet I still have to spend about 30 minutes a day convincing myself that I don't NEED a BHSE to squeeze the last ounce of performance out of my OII. (I think I am losing the battle.)

I too am a science minded person, and regarding #2, I believe I would have my mind changed by the result, whether positive or negative. I've never tested cables, software players, sample rates, or even DACs, even in sighted testing, so I'm positive I have lots to learn about my preferences regarding gear. Honestly, it would be a relief to know one way or the other.

khaos · « **Reply #6 on:** April 13, 2012, 11:01:08 AM »

Quote from: Analixus on April 13, 2012, 08:02:27 AM

Financially, it does imply a separate room in a meet, otherwise enough people to lend the gear and someone willing to crunch numbers, otherwise, this is the purpose of this thread.
More than that if we agree on sample gear familiarity. I'm including sources and amps in question so loaners/acquisition and familiarization time is a complication.

On the other hand, Fletcher-Munson has nothing to do with it, with a general volume control (perhaps the 32 bit VC of Foobar on PC) placed upstream to all the amps, you set the volume of all amps simultaneously, so any difference heard between the amps would not be due to volume differences.
Right, I was just concerned w/ variation among test subjects, not gear. If they have master control it's moot.

Make it anonymous?
And unpublicized or widely circulated, no privacy sacrifices w/ youtube vids.

That's an issue only if the result is negative (no difference between amps), in this case, switching is as simple as taking off one pair of HD 8000 and putting on another pair.
To elaborate, the track would ideally be reset if headphone switching is used and the idea was to limit or control ear/brain fatigue and psychoacoustics.

Did he manage to do so?
Someone was scolded. It wasn't me.I think the line was something like, "are you using MP3's to try to f**k with me again?" His chain is just too transparent.

About familiarity, Iwould say that if you could get a dozen or more people in that testing room, it would be quite a success, but you can't expect everyone to be familiar with every piece of gear, however it may be interesting to see whether there's a larger variance in the evaluations depending on familiarity.

I don't understand you second point about volume.

Of course, no youtube videos, unless they request it

About track and headphone simultaneous switching, good idea.

Purrin the trickster, may even be a good cartoon name

Anaxilus. · « **Reply #7 on:** April 13, 2012, 07:33:01 PM »

Quote from: khaos on April 13, 2012, 11:01:08 AM

I don't understand you second point about volume.

Only that each person has a different contour and preferred listening level for critical analysis. The listener must be able to define it prior to commencement of the test rather than a uniform level established for all participants. Gain matching for all gear is a given, but universal gain matching for the entire pool is a flaw in design.

As an aside, it's easy to see why most of the oft cited tests yield 50/50 results. Their poor designs inevitably lead to the subjects getting frustrated and doing random guessing. No surprise considering the agenda of some of the test designers.

Last night I was thinking how tempted I am to run a simple, verified test on myself:

Setup 1: Clip+ (set to '0') > O2 > HD800
vs.
Setup 2: PWD > BA > HD800

I'd feel 95-100% confident nailing that but then random naysayer X would protest:

'Of course! Tubes suck and produce noise, durr! You just like coloration and are too ignorant to know it, derp.'

'But the tubes sound more transparent and detailed? How is that?'

'Derp, you're an idiot who likes to spend money, that's not possible, derp. It's all beyond audible levels except for the noise you prefer, herpety derp.'

'But why do I hear more layers and data from my source material?!'

'Shut up idiot! Placebo and mental distortion from the tumor in your brain!! L2 Science! Derp.'

So, umm...yeah, the point again would be?

ocswing · « **Reply #8 on:** April 13, 2012, 08:27:58 PM »

Quote from: Analixus on April 13, 2012, 07:33:01 PM

So, umm...yeah, the point again would be?

Derp.

Marvey · « **Reply #9 on:** April 13, 2012, 08:47:30 PM »

WARNING: STREAM OF CONSCIOUSNESS POST

When we talk about perception, we are starting to get into philosophical and metaphysical stuff. Gross elements of perception can be measured. Subtle elements perhaps. I've never been a reductionist anyways.

While I fully understand that emotions, moods, money-spent etc. may heavily weight upon perception, I also believe that we can arrive at a consistency of perception over time (an average of differing moods, thoughts, etc.) The true test for me is stuff I like to spend time with. This is easier for me since I know exactly what I want. This may be more difficult for others, especially those with three-four different systems and eight headphones. (I tend not to trust those guys' opinions, because they obviously like a wide range of really different setups.)

Science is only a tool. It does not explain consciousness or the nature of perception. Current measurement methods perhaps only measure at most 50% of what we hear. That being said, there is a high correlation between headphones which measure like shit and sound like shit.

But we are going to do this, perhaps expanded criteria should be used rather than just bass, mid-range, treble, clarity, detail. Musical reproduction involves much more:

Immediacy - an overall bucket of how real or true to life the recording sounds or feels.
Dynamic capacity - IMO, never measured, but where audio reproduction fails most compared to real-life.
Compression - having the softest sounds be soft and loudest sounds be loud.
Granularity in volume level.
Resolution - not be be confused with "detail".
Articulation / transient response / blackness between notes, sounds, etc.
Clarity - related to 4.
Noise - related to 5
Granularity in sound going forward in time. Grain or liquidity.
Ability to reproduce softest sounds during loud passages - related to 5,6,7
Distortion, related to 6,7,8
Tonal balance.
Apparent bass and treble extension - related to 12.

People hear differently and have different criteria. The criteria above are the ones that I use. Most people may not give a rats ass (or even a thought) about many of the aspects above list. I guess what I'm saying is all this complexity, worry, and attempt at being objective with subjective impressions is unnecessary: I learned to trust my observations, and those observations of whom I trust.

Among people who I know well, LFF, Anax, RD, CEETEE and I all "mysteriously" happen to hear the same exact things with gear, but yet at the same time, we have different tastes. It very may well be that we are more effective at communicating with each other. Or it could be that we don't put so much of our egos into it, and are willing to admit our preferences - and that our "stuff" is not necessarily perfect, better, or best (then again, I still feel the Leben sucks.)

Author Topic: Using a meet to do some blind testing? (Read 1079 times)

khaos

Using a meet to do some blind testing?

Anaxilus.

Re: Using a meet to do some blind testing?

khaos

Re: Using a meet to do some blind testing?

raif

Re: Using a meet to do some blind testing?

Anaxilus.

Re: Using a meet to do some blind testing?

Maxvla

Re: Using a meet to do some blind testing?

khaos

Re: Using a meet to do some blind testing?

Anaxilus.

Re: Using a meet to do some blind testing?

ocswing

Re: Using a meet to do some blind testing?

Marvey

Re: Using a meet to do some blind testing?

(adsbygoogle = window.adsbygoogle || []).push({});

Author Topic: Using a meet to do some blind testing? (Read 1079 times)