Hello;
If the rotor moves in the wrong direction when applying torque, you should reverse the motor cables. If it oscillates around the rest position you should work on the motor tuning (PID). If it moves in both directions but does not stop, it is getting noise on the step/dir signals, there is a problem with the encoder signals, or there is a problem with the output stage Mosfet driving, as pointed out by Henrik.
Regards,
Kreutz.