The facemill will probably be worse.
You do not mention depths or anything but I am guessing the tube you are boring is just sticking up from the rectangular tube which is clamped down onto the table. This gives the opportunity for a lot of flexing both in the walls of the tube being bored and in the face of the rectangular to which it is welded. It is also almost certain you are using an insert cutter which is negative rake and these cutters exert a lot of force against the surface being cut increasing the flexing. And to compound the problem the flexing can be greater in different directions around the tube depending on whether it is directed across the reactangular tube or along the rectangular tube. The result is every rotation of the cutter make the tube flex and return and vibration builds up.
What to do?
You need to stabilize the tube being bored and one way to do this is make a wooden collar cut to fit the tube with a bit of stiff foam rubber between. Make tis collar out of pieces large enough to extend beyond the ractangular tube, clamp it on and then put packing pieces between it and the table at either end and clamp them down. The reason to use would is that it absorbs and vibration that does develop, the foam rubber helps with this, and wood is not straong enough or stiff enough to move your workpiece out of position when the collar is clamped down.
A quicker method which may not be feasible because of your tube size, is to wrap strips of lead around the tube and overlay them with electrician's tape tightly wound on; lead solder works great for this. But for your size you would need many feet.
An open mind is a virtue...so long as all the common sense has not leaked out.