"I kind of wonder about your "Hooked up a 1k resistor and red LED.." - If this LED wasn't a fast response part*, could this be the cause of the problem?" - FredM
Just tried a 220R and am seeing essentially the same spectra. I think if there were any digital issues at the serial interface they would show up as gross issues with the spectra. I'm actually kind of shocked the results are as good as this, with the second harmonic ~80dB down (the sine waveform has its LSBs simply truncated to fit 16 bits, rather than noise shaped or dithered). What I don't get is why feeding it a 0dB digital signal causes severe clipping, though a quick web search seems to indicate others have perhaps similar problems with SPDIF levels:
http://www.kemper-amps.com/forum/index.php?page=Thread&threadID=4722
http://duc.avid.com/archive/index.php/t-43770.html
It's interesting that the value of -11dB mentioned in the first link is suspiciously close to the >> 2 bit / -12dB case I'm seeing. I have the huge advantage in that I know exactly what information the serial SPDIF contains. Those poor slobs are shooting in the dark behind a wall of drivers and other SW, trying to make sense of things. Yet I still don't have a clue. The data should be aligned so that MSB is in the MSB location, and any unused LSBs set to zero.
I see a Toslink output on my PC motherboard but not sure I've got the time nor the patience to see what's going on with that.