Well on my table (homebuilt but I have the same controls as you) I had the same issue.
1. Move the motors away from the rack and run a program. Unless you have some neat gear you need to tune your motors using your hearing and touch. Some dithering at a standstill is normal, but when running the program the motors should run smoothly. At first my y axis was herky jerky in the slower parts of the arc.
2. The spring on my x axis wasn't strong enough and the pinion would walk out of the rack on accel and decel, causing more roughness. Also make sure the pinion is parrellel to the rack.
On all equipment there are 2 levers...
Lever "A", and Lever F'in "B"