i guess you do know that points gap changes result in ignition timing changes .. setting gap is the only way to set timing since the stator plate can't be moved. If adjusting point gap (and timing) made a difference, its a clue.
Most engine castings are rough .. some worse than others. Plans on the engineer's drafting table call for smooth cast surfaces but that aint gonna happen in production.
Cleaning them up and matching port openings is always a good for the engine even if the results are not so dramatic as to be felt in the seat of your pants.
I would not mess with the transfer port roofs (as if you could find some tool that can get up there) since their shape is critical for scavanging..