I suspect your machine is just too mini for the job. But, try slow spindle speed, and shallow depth of cut, and relatively steady but coarse feedrate. You didn't describe your cutter diameter, or the depth of cut and width of cutter engagement. Generally on machines without ballscrews, conventional milling is safest. This method prevents the cutter from "hogging in" by the backlash amount, which it will certainly do if it is climbing. Once it takes that initial monster chip, everything goes to hell real quick and its hard to tell after that what went wrong
First you get good, then you get fast. Then grouchiness sets in.
(Note: The opinions expressed in this post are my own and are not necessarily those of CNCzone and its management)