It's a bit relative to your sweep distance too. If you are sweeping a 3" circle (like maybe a vice bed) I might be a little more fussy but if you are sweeping a 6" arc then you are doing real well at 0.0005". Also got to remember that these little mills are going to flex. I can tell the tram difference in a heavy cut (0.050") with a 2.5" face mill vs a cleanup pass at 0.002", the heavy cut flexes the head enough to get it to lean a bit downward at the side pushing through the metal and only shows cutting at the leading edge, the cleanup pass shows just a hint of tool marks at the backside of the cut. Longer tools do it more than shorter ones.
Sounds like you have it nailed down pretty well. I'd call it good and go on.
CNC: Making incorrect parts and breaking stuff, faster and with greater precision.