It is likely not your nc code. Tip wear tends to have an accumulative effect. If the tool advances by .100 in depth for each pass, it is really only the flutes of the tool near the tip that are taking the brunt of the wear from usage. When they get slightly dulled, then the tool deflects and the sharper flutes further on up have no chance to ever remove what they could remove if the tool was not deflecting.
Rough drill the material out of the hole so as to leave only about .005" to remove with the endmill. Then using a brand new endmill, interpolate the hole at depths of maybe 1/4 to 3/8 inch, removing those last few thousands, going around twice at each level. Consider how little movement is actually going on with a .1875" tool orbiting within a .254" hole. 5 ipm @3000 rpm is too high of a feedrate to actually give the flutes a chance to remove all the material. Because the circumference of the circle is only about .0625*Pi = .19", at 5 ipm your tool is completing one orbit in about 3 seconds. Because you are using about 50 line segments to make an orbit, it is doubtful that the control can force the machine to make each endpoint positively before the next movement begins, so this sort of mushy movement results in what I call 'lazy interpolation', where the machine lags behind the programmed motion. Try 1 ipm and see if it is any better, or indeed, a rounder hole. It is difficult to know how tight your machine really is, and you might actually have a 4 cornered hole at the end of it all. Then, you get a custom reamer to really solve the problem.
First you get good, then you get fast. Then grouchiness sets in.
(Note: The opinions expressed in this post are my own and are not necessarily those of CNCzone and its management)