At least regards overall phase, it seems I personally can't tell the difference. However, I imagine that relative phase is quite important for sonics. For example, imagine we have a source emitting two pairs of sinusoidal tones, the first pair is centered at 100 Hz and separated by 1/2 Hz, the second pair is at 1000 Hz and separated by 1/2 Hz. Within each pair, the frequencies are close enough for your brain to perceive the interference, while the two base frequencies are far enough apart that your brain ought to hear them separately. If they are all in phase, then one will hear two tones, 100 Hz and 1000 Hz, each beating in amplitude once per second and doing so in perfect unison. If the phase difference between the first pair is different from the phase difference between the second pair, one of these beating tones will be temporally offset with respect to the other, and this would definitely be audible. This could certainly be the case with a really crappy phase response. Although, I suppose the lesson learned from this example is that the only thing that matters for such timing is that the phase response not differ too quickly over small frequency ranges. So, perhaps then all that matters is that the absolute value of the derivative of the phase response is sufficiently small over all frequencies. I imagine that this is probably true of most phase fuckups, and thus why we don't hear such glaring issues in timing in practice.

On the other hand, I have heard a bit about the dependence of concepts like soundstage and imaging on things like fine details of phase, timing, and so on. So, now I'm wondering, could all these phase fuckups introduced in EQ'ing, recording, and even at the headphones be together responsible for a good portion of why most recordings fall short of sounding truly 'realistic'?