Brush DC VF2 backlash like problem, X axis drift

Started getting what looks X axis drift towards (-) on an old brush motor VF2. Milling a pocket with sideways hatch and finish pass, finish pass on (+) side does not line up. X axis drifted to (-)

Jogging with an indicator: will jog X off indicator and back with no apparent backlash

- When another move back and forth is added: first will not return to zero but 2nd move will
- Multiple steps back and forth only the last will return to indicator zero
- Middle steps miss zero position by upto 0.020
- Reducing feed rate reduces X (-) error on middle steps, last one still returns to zero
- Adding a dwell (any amt of time) after each X move eliminates the error completely

Haven't removed the way covers yet to check ballscrew by hand.

Any idea what might be causing this?

Seems too consistent to be bad motor brushes or encoder problem. Maybe motor-ballscrew coupler? Dwell allows it to coast back?