Dan, I've had the same idea, but mine was inspired by radial drill presses. If you're looking to use a rotary table, however, I think they're plenty stiff for wood and metal milling.
I, too, think the problem will be software, not hardware. If you can find CNC controll software that will control it and CAM software for it, I'd say give it a go.
My name is Electric Nachos. Sorry to impose, but I am the ocean.
http://www.bryanpryor.com
(Note: The opinions expressed in this post are my own and are not necessarily those of CNCzone and its management)