I don't know the QT50 well... but the tranny oil is not supposed to be all the way up to the top.
So.. drain it.. and then look carefully on the side of the cover down by where the filler is.. lots of Jap bikes say right on them what the capacity is supposed to be.. like 400ml.. or 400cc (the same thing.. 1ml equals 1cc).
If not..call a Yamaha dealer and ask them (or go visit one).. or try 400 to 500 cc.. that will be about correct.. (you should still get the correct capacity from a manual or dealer or owner with a manual).
And ATF or 10w30 motor oil will both work.