Right after getting ready to drill, my brother-in-law suggested I try some thick 1/4''-1/2'' nylon washers to push out the bracket so that it would bring the holes closer to proper alignment.
The washers did help, but I had to apply moderate pressure on the pipe while screwing in the bolts.
Not sure if this is a long term solution.
