You are absolutely correct. I am picturing things in my head but I do think I need to consider different material dimensions and then put it all on paper. What I have "pictured" could be completely different than reality lol.
I do have some heavy steel tubing that I was considering using for the spindle shaft housing (not sure if that's the correct term). It's approx 1 5/8"od and 3/4id. I was thinking about using some sleeve bearings on the inside and then running the spindle shaft through it. I would think that would give me a significant amount of rigidity. Hopefully that makes sense!
Besides those factors do you see any other real issues with the idea? Another option that I was considering is one of the milling tables for a lathe. Mount the T table to the column and the spindle to the base. Below is a link to one option.

https://www.amazon.com/dp/B077GP8RN8...v_ov_lig_dp_it