I know these types of issues can be frustrating. Out of round can also be Mach3 settings (IJ, CV Backlash etc) but the jagged edges look more like a mechanical issue. Incorrect drawing (how the cad is drawing segments, arcs and circles) can cause issues also... especially how your post processor handles the output. Specifically how it writes the gcode to cut arcs and radii.