Should be ok, but here's my thoughts
You shouldn't be able to walk bearings in the cases if everything is cut right. When the cases are assembled, the bearings should get basic real close to dead ended into the end of the machined race in the case. If, when you put your bearings on, the space between the furthest out edge of the cases is too large and the pressfit tolerance is a little to spacey, then this is what happens. Next time do lots of measuring and shimming or get better cases.