I have the same mac mini connected to beosystem3 and i have the same problem...Try lower resolutions.Or change picture settings of the monitor.I came to realise that When i set resolution to 1080p, the picture setting on my tv(samsung 52lvxxxx four year old lcd) must be on "just scan" not "16:9" or "zoom"...
BL5 (pair), BL3 (pair), BL7-2, Beosystem3 w/dvb-s, cabinet, BS4 w/DAB, BV8-26 w/sys module, Beocom 2, Beo5, Beo4, Beoport, A9 and a garbage DVD2