A little bit of rust falling is no big deal. Screens and filters take care of that.
Assuming it's rusty inside, the rust inside the cap _can't_ get out. And thats the problem because rust clogs the cap's vent. Air has to be allowed into the tank to replace gasoline or fuel flow will stop.
i had an annoying performance problem.. like the bike was running out of fuel.. All of a sudden it would slow down for a moment and then, just as suddenly, everything was OK.. this went on for a week.
i traced the problem to the gas cap.. Understand that i had already very carefully cleaned the cap, run acetone through it and poked wire in the vent holes and blown it out with compressed air and all that.. What i didnt know was that the cap vent kept getting clogged up due to the internal rust.
The clogged vent produced a vacuum in the tank and fuel flow slowed to a trickle.
After opening and sandblasting the inside of my cap, the bike never had that problem again. Fuel now flows from the tank's hose in a steady thick stream.
btw, for a rubber gasket add-on as suggested in the previous message, use a piece of tire inner tube.. maybe two.. Whatever it takes to get a snug fit.