Yeah, there definitely is a too much point, probably when the balls are no longer contained safely in the ball...uh...court.
2mm actually seemed like a lot after I marked it out and did little by little, checking the ballspace periodically. I would guess I took off about 1.5mm max, and the balls are in there good still. Have yet to fire it up.
Of course my back plate warped a little when heating it off, and the "plastic" cage the balls sit in melted and bubbled quite abit too, but it doesnt seem like it will affect performance, as the melting was just at the base of the cage, not interfering with ball movement. I hope the warping doesnt affect perf too much.