John:
What you propose is a VERY bad idea. Damage will surely come to one if not both receivers.
What you want to achieve can be done with a speaker switch box (but not just any.....get some help with choosing the right one). If you have basic soldering skills, go to an electronic supply store and buy a double pole double throw DPDT switch. You will need two switches one for each channel.
Derek