Use a lathe. Or hire it done on a lathe. This permits you to open the hole with a smaller drill, rather than requiring super torque, and a super tool to go along with it. Plus it allows you to do a nice accurate finish bore without a lot of fooling around.
If I had to do it in a larger part, I might open the hole with a drill, and then use circular interpolation to enlarge it on the cnc mill. This method works okay, but does tend to use up a few more inserts than the lathe would.
First you get good, then you get fast. Then grouchiness sets in.
(Note: The opinions expressed in this post are my own and are not necessarily those of CNCzone and its management)