Obviously you are losing steps, I would imagine.
Is there any reason to expect that more torque is required to drive the Z down than to raise it? How does everything feel mechanically when you turn the axis by hand?
Are you plunging into the work? What kind of materials do you cut?
Have you run trials where you simply run the Z axis up and down in the air for a while? Does it lose position?
Have you tried homing the machine after every Z plunge?
Sorry, I can only ask questions That dilemma is why I would never run open loop feedback on a machine tool.
First you get good, then you get fast. Then grouchiness sets in.
(Note: The opinions expressed in this post are my own and are not necessarily those of CNCzone and its management)