Very nice CNC conversion.
On the BLDC motor, I suspect the controller is the main problem. Before changing the motor it might make sense to try a quality BLDC controller. I suspect the motor is capable but the electronics aren't doing a great job with it.