The reason you cannot grab any power amp and hook up any headphone directly to the speaker terminals has to do with impedance, power rating of the HP and output VOLTAGE.
A purely technical reason so to speak.
For this reason in the old days (when amplifying parts where relatively more expensive to resistors) it was common practice to drive headphones from the power amp (and with an intarnal switch in the HP socket that switched off the speakers) via a single resistor or a resistor divider network (2 resistors).
Nowadays, as already mentioned and opamps are cheap mostly a 'dedicated' HP amp section is used.
Inessence a 'C'Moy' is inside.
I prefer to call it a 'standard application' as a C'Moy is nothing more than the basic schematic found in almost every datasheet/application note and is not an invention of Chu Moy, he just used that standard application and mounted it in a tin can with 1 or 2 batteries.
Brought him even more fame...
Adding 1 (or 2) resistor(s) is done for several reasons.
The biggest one is to protect headphones from blowing up (respecting the power rating of most headphones being around 0.2W)
a power amplifier delivers an output VOLTAGE and must be able to provide the CURRENT the load 'dictates/asks/draws/demands'.
With amps being able to deliver current to 4/8 Ohm speakers it is easy to deliver currents to resistances higher than that as the current drawn will be much less as when speakers are used.
I will humor everyone and start throwing POWER figures for headphones for 2 amps.
a 15W/8 Ohm amp and a 60W/8 Ohm amp.
Since this is a factor 4 in power you can do the math for a 240W/8Ohm amp.
On ear headphones in general have impedances between 32 Ohm and 300 Ohm so these will be in the example.
15W amp ... 32 Ohm ..... 300 Ohm (headphone impedance, not output resistance)
power: ......
3.8 W .........
0.4 W max output voltage: 11V
a 32 Ohm HP with 0.2W rating runs a risk of being fried
!
60W amp ... 32 Ohm ..... 300 Ohm
power: ......
15 W .........
1.6 W max output voltage: 22V
So that is the main reason power amps are not suited.
Also they in general have more noise (as already stated) and thus also limited dynamic range for small signals.
Also as mentioned the gain is relatively high and thus turning up the volume knob results in screamingly loud levels when barely touched (true for low impedance headphones, somewhat less for higher impedance HP's).
But the main reason it should not be done is the... rated power of the HP.
There is a cheap and easy fix though and that is using a series resistance.
Common values were between 100 and 600 Ohm and this also explains why those oldie amps often sound so good out of their HP sockets.
I have touched on that subject in my articles about power and impedance.
same amps but with a 100 Ohm output R added in the 15W and 330 Ohm in the 60W
15W amp ... 32 Ohm ..... 300 Ohm
power: ......
0.22 W .........
0.22 W max output voltage: depends on voltage division
60W amp ... 32 Ohm ..... 300 Ohm
power: ......
0.12 W .........
0.35 W max output voltage: depends on voltage division
This could be 'better' figures when a voltage division network is used (which was often the case)
It also shows on some HP's (due to the higher output R) some headphones can sound considerable more 'mellow'.
Also this voltage division adresses the gain issue (volume control is more usable) and noise is lowered, dynamic range increased.
So there you have it, the real reason it cannot (should not) be done with low Impedance HP's, unless rated for several watts such as orthos and some monitoring headphones that can 'double' as small speakers on the console.
High impedance headpones 300 /600 Ohms CAN be connected directly when power ratings allow.
Certainly to the lower wattage amps.
Since orthos have a lower efficiency and high power rating there is little to no objection powering them direct from speaker terminals.
Watch out with the power though (amps above 40W)