Tool moving
One of my customers has just been fighting a similar problem! He was using a nice new 6mm ball nose cutter, but was loosing position vertically. In this case it was the mount for the Kress spindle that was moving very slowly down the V plate. But it does show that the cutting pressure puts some strain on the vertical axis. Adam just nipped the mount up a little more and things came good, so it is more than likely that you have a similar problem and it's worth checking both collet, and the head stock.
On a slightly different tack, it is worth pointing out that as the tool gets blunt, then the weight of the motor ( the Kress is a lot lighter than the ER16 assembly ) is not enough to hold the tool down, and so backlash on the Z axis is as important as X and Y
Lester Caine - G8HFL
http://medw.co.uk - Home of electronics for the Model Engineer