If the internal spacer is longer than where the bearings seat, it'll push the bearings side to side, from braking, turning, side loads, etc.
You could probably just cut the spacer down, and make up the 3mm gap with bigger outer spacers.
I have a few lelue hubs and don't have that problem.
Do the bearings fit tight in the hub? Maybe the hub shell is a little worn, causing the bearings to rock a bit?