Didn't meant harsh in the context of sound quality. More like a harsh environment for signal integrity.
TOSLINK I think typically uses multimode fiber which suffers from internal light reflections which effectively filter the light signal and restrict it's bandwidth and reach. Futhermore, most audio equipment seems to use LEDs which might compound on the dispersion problems. Now SONET stuff typically uses lasers and single-mode fiber, but we are talking $ there.
Like you said, proly different analog/digital interfaces which might yield different results.