the 9x20 lathe is an extremely flexible package and by that I mean not rigid. there are ways of making it stiffer. I have in the past taken way more than your cutting on my 9x20 lathe however there are plenty of areas that need to be addressed.
1. spindle preload. make sure you preload the spindle really well. if its flopping around you will have no hope of cutting anything
2. carriage stiffness. make sure that the carriage clamps are tightened and that its not flopping around
3. cross slide stiffness. this has a pretty crappy gibb, so make sure that you tighten all the screws and they are making good contact.
4. make sure your spindle is aligned to the ways and not cutting a taper.
5. make sure your tool nose is on center of just below.
these are just a few of the areas that I have found that need to be addressed. also I didn't see what HP vfd and motor you have, but your spindle belt maybe slipping under higher load.