You're right, they don't pop off like many of the others. This will take a bit of patience, but it isn't too hard.
Russ your hand down the side/edge of the grills and you will feel a series of openings (6-8 total I think, but don't quote me, you'll definitely feel them). You will have tp push a finger into each opening to release a plastic tab which is captured in the opening. As you press into the opening, you will want to slide the grill forward to prevent it from recapturing the tab as you move along to the next opening.
Take your time, and remember that you can always replace the grille when you order new drivers.
Russ
We kid because we love.
Bang & Olufsen Tysons Galleria
McLean, VA USA