If the toolholder is good, and the spindle taper is good, you might have broken springs in the tool retension rod. When the tool gets enough bite on the work, the helix of the flute is able to pull the toolholder out of the spindle a wee bit. This would be visible after some extended period of time as a brown rust stain on the tool holder shank, called fretting corrosion.
First you get good, then you get fast. Then grouchiness sets in.
(Note: The opinions expressed in this post are my own and are not necessarily those of CNCzone and its management)