Get some tracks you are extremely familiar with. Hopefully something of at least decent quality and that can stress the headphone in nearly all aspects for a "general" test. Then include some more specialized tracks to listen for certain aspects (soundstage, vocals, fart timbre, etc.). I do a mix of quick A/Bing (10-30 second segments) and ~10 minute sessions before switching to the other headphone (also consider that some headphones have pads that give the best sound after warming up for a few, brain burn-in, etc.). The quick tests will help with immediate, specific differences, and the longer tests will give you more of a general overview on the differences.

I think measurements are also great at helping one identify what they're hearing, unless they're crappy measurements. Boomy, bloated bass? If measurements show high harmonic distortion and/or an elevated bass response, it could be that. Or, for something like the HE-400, measurements helped identify why (at least one revision) sounded both dark and bright to listeners at the same time. Or, as kunlun mentioned, it might help you identify specifically where a headphone is ringing. I have found after a while, you will start to pick up on things and be able to identify what you're hearing down to a pretty specific, quantifiable range.

The more stuff you sample, the easier it will be to figure out where your preferences and sensitivities sit compared to most others (again, measurements can help us more easily figure this out if used correctly). Understanding that frame of reference helps when communicating your thoughts with others.

I don't want to get too off topic, but does anyone have recommendations for a good recording of a piano? I'm most used to listening to piano as the person playing, and I've not yet found anything that adequately reproduces that experience. Granted, I can't say I've really looked either!

Here are a few tips I can offer from my own personal experience:

-Develop a collection of test tracks. You can find CDs and downloads that are marketed as audiophile tests for this very purpose. However you can also use music you're familiar with too if there's something about it that stands out, for instance a track with a ton of deep bass or one that has a lot of left-right panning in it.

-Some audiophiles poopoo so called "modern recordings" of rock and pop, but these can actually be very helpful in highlighting a headphone's problem areas. When you go to trade shows, you'll often hear the same popular selections of girls strumming guitars and small ensemble jazz groups. Nothing wrong with either---I love both---but these recordings tend to sound good on anything. That's why manufacturers love them. If a recording is entirely midrange, it's not going to set off any of the potential problem areas for headphones which tend to be at the frequency extremes: the lowest bass and highest treble.

-Field recordings are very useful for determining a headphones spatial characteristics (soundstage, imaging and placement, separation). I'm actually a little obsessed with field recordings and think they're really interesting.

-The reasoning behind direct A/B comparisons, the back and forth listening between two headphones, is just as you say: our audio memory can be unreliable at times. That's why folks in this hobby will often add a disclaimer if they're recalling something they listened to a while ago.

-That being said, there are factors to consider with direct A/B comparisons too. Personally speaking, I find my perception of how something sounds can change from one day to the next depending on my mood and what I've been listening to prior to that. For example if I've been listening to the FitEar MH335DW customs, the UE Reference Monitor sounds a bit thin for a while when I switch to them. Alternatively if I've been listening to the UE Reference Monitor, the FitEar sounds a bit thick and sluggish for a while when I switch to it. In other words: your brain needs time to adjust.

-When doing a direct comparison, make sure you're using the same gear upstream. Contrary to that popular mantra: amps don't just make things louder, they do influence the sound. There is no true "wire with gain" in my experience. Using the same source and amp allows you to have a baseline and focus on a particular variable.

-Personally, I find rapid fire comparisons back-and-forth to be tedious after a while. After trying to be "rigorous" about such things for a few years, I realized something that was originally a hobby had become a chore for me. These days I favor a longer term comparison. I prefer to actually live with a piece of gear for a few weeks, allowing myself to acclimatize to its characteristics and get a feel for how it sounds on a variety of gear and in different conditions. Additionally synergy is important in achieving an ideal sound: finding the right match for a headphone in an amp makes all the difference. I try to report on these things as they occur, yet another reason as to why I favor a journal style of impressions.

-Ideally, I come to develop a profile for a particular headphone. Measurements are helpful in understanding the overall character of a headphone, but experience is needed to understand how that translates to living with a headphone day to day. In other words: measurements never "speak for themselves," they have to be interpreted by an individual with his or her own preferences and priorities.

-As part of this profile of a headphone, I like to develop an understanding of how it performs on a variety of gear. For instance how it performs out of a portable Walkman on the one hand and on the other how it scales up with higher-end stuff. This also helps me get a feel fo r synergy. It allows me to compare headphones under "real world" conditions, in light of my changing moods and every-changing preferences for certain genres.

-Ergonomics and build quality are also important to gauge over time.

-That being said, I find using the A/B method is still helpful when I need to "zoom in" and fill in the gaps of my overall understanding. Like when I want to refresh my memory regarding something very specific.

-Trust your ears. The human brain is an incredible analytic tool, the scientific device par excellence. Measurements are helpful as a means of understanding, framing, and presenting your thoughts to others. However measurements still must be translated into language. Subjective impressions should always come first.

-Yes, we are all susceptible to bias. I don't think this is such a big deal so long as we have a general understanding of those biases. Often times folks who are adamant about pointing out the bias of others miss their own. We all have our own reasons for doing this and priorities when it comes to listening. Know what YOU want out of this hobby and communicate it to others as best you can. There's nothing wrong with liking a highly coloured headphone. Just don't try to pass it off as neutral.

Well, I really was just looking for excellent recordings, but I was always particularly fond of Beethoven's "Moonlight Sonata" and "Fur Elise" or perhaps Kabalevsky's "Sonatina No. 1 (Op. 13, No. 1)." I also had a lot of fun with Bach's invention pieces, particular 8 in F Major, but was more "moved" by the other pieces I mentioned (Beethoven is all about dem feelz if you wants to play it right). I also liked some Debussy pieces, but seeing as though I never really learned to play them, I can't recall any by name.