Machiel:
After removing the batteries, the display turns off directly.
Hmm... that symptom sounds like the remote might actually keep transmitting, as if a button was stuck down.
Look at the LEDs with a cell phone camera to confirm that they go off when you aren't pressing any button.
I
don't think replacing the caps will do much. You should make sure that
the wiring from the batteries (and their solderings) are good, and if
there is any signs of battery leakage or whatever spilled on the circuit
board, it should be cleaned.
Unfortunately the circuit boards weren't really designed to be serviced, there aren't any schematics around for example.