One technical reason I can think of is that the power supply of most televisions can run in just two modes: standby mode that keeps the CPU and remote control receiver running (and can't do anything else), and "full blast" when the high voltage supply and picture tube are on. Using it for just audio would require something in between. Most sets just aren't designed to do that, short of some from our favourite brand: they have a separate audio only mode which runs the power supply in reduced output mode, and only turns on the audio amplifiers and signal routing.
I did have a domestic TV set (Salora) way back that had an optional FM tuner. Worked just fine, but it felt a little stupid having the tube whistling on with a black picture while listening to radio.