I advice to set the spindle to the highest possible RPM.
You know on industrial PCB drilling machines there are air spindles with really high over 50k RPM.
Probably you don't have an air spindle at home, but a 20-30k RPM would be good to have.
What diameter of holes are you drilling?

In general if the drill bit has low diameter, the machine construction should be robust and vibration free, because a small buckling of the Z axis may break the drillbit.
Also too low feedrate of the drilling in the material may cause a burning and damage of your tool and a too high feedrate may also damage the tool, so try to find the midway.