Seems like it would allow the pipe to flex along with the swing arm and eventually it would fail somewhere. However, every picture of a stock VA shows the muffler secured by a clamp mounted to an arm extending down from the axle nut or occasionally from the lower shock mount.
Later models, like mine, have a separate arm extending rearward parallel to the swing arm to which the muffler is secured. You could go this route. You would have to drill two holes roughly horizontally above the center stand mounting point on the frame and attach an arm, either flat stock or tubing with flattened ends.
Put your muffler on the pipe first to determine the angle you need for the arm then drill the holes accordingly. It would be good if you could allow some fore and aft adjustment for the silencer, because its position can affect performance and you'll want some trial and error runs to figure out its best position.