I'm not sure I understand your approach, Kong. Why didn't you just do the milling on your lathe? You could have rigged up a link from the tailstock to serve as a positive feed in the Z axis. A milling vise attachment (built for lathe usage) takes care of the vertical Y and the normal cross slide takes care of the X.
See my ad in the for sale section
First you get good, then you get fast. Then grouchiness sets in.
(Note: The opinions expressed in this post are my own and are not necessarily those of CNCzone and its management)