iTunes at 320kbps and lower will seem harsher if you have a full resolution reference and are playing back on good speakers.
You might want to check your settings for iTunes, though. Make sure you have no equalizer, volume leveler, sound enhancers or anything of the like enabled.
After having spent a long time now listening to lossless files, playing back through optical out to either a receiver with a DAC or a stand-alone DAC (or straight to the digital IN on my BL5s), I am no longer able to listen to low resolution files, and particularly not from the analog out on my Macs.
My ears have gotten too attuned to the difference in quality and I pick up on it, particularly with acoustic music. As you say, there's a harshness, a lack of body, that just doesn't set with me.
I don't quite understand how you'll be using the Apogee Duet, though. It's primarily intended as a recording interface between microphones, guitar, keyboard - to the Mac.