If you are going to say speaker amps are better for headphones than headphone amps, you really need to say which amps you are comparing directly.  General, ambiguous observations don't really offer any value at all.

e.g. I was told V-8s are faster than V-6s.  However, I drove a V-6 (Nissan GTR) and it smoked the V-8 (Chevy Silverado). 

See what I mean?

Plus, modifying the FR via damping factor tells us about a sound signature/balance but nothing about the amp's actual ability.  You can achieve the same affect on pretty much any amp.

There are some excellent speaker amps but they are not usually cheap.  Headphone amps by nature must generally cope with exposing all sorts of issues in closer proximity to the listeners ear via a headphone.  Speaker amps can let more stuff slide as 'issues' more noticeable on headphones would be diffused across a room.  This is not a absolute rule, just a general design consideration based on primary goals like power delivery. 

Sports cars are not required to have only two doors, you can try four if you like.  Sometimes a five-door wagon is even better than a four door!  However, no doors are the best choice.