As awerby said, it may be lost steps, however if it's happening consistently when changing to an op with a twist drill, your problem is most likely that the column is not perpendicular to the x and y axes. I had a the same issue with a larger machine - see my thread linked below for more detail.
https://www.cnczone.com/forums/bench...ml#post2416552
If the column of your mill isn't perpendicular to both the x and y axes, as you change to a longer tool (for example a twist drill) the head of your mill will move up, but it will also move side to side/front to back depending on what angle the column is set at. This gives you the 0.005" to 0.01" offset in hole location. To eliminate this, you'll need some sort of square and a dial indicator - set it up on the table and jog the Z axis up/down along the square and shim the column respectively until there is no more movement in your dial indicator. Do this with the square oriented in both directions (x and y).
-Adam.