You will find it best using the lowest PWM frequency practical.
You probably have PWM distortion, where the rise time and fall times are not the same somewhere in the PWM circuitry between Mach3 and where it is decoded to speed.
Ordinary 'standard' optos are not up to the task.
I use a 10 Mbaud device to overcome PWM distortion.
It is a device used in communications, and has equal rise and fall times, and exact rail to rail output.
A simple open collector trying to integrate with a resistor and capacitor just won't work properly. Charge impedance to capacitor must be the same as the discharge impedance, for starters, and it needs to be CMOS rail to rail drive.
Or simply use a microprocessor, like a PIC to decode the PWM accurately.
See my PM.
Super X3. 3600rpm. Sheridan 6"x24" Lathe + more. Three ways to fix things: The right way, the other way, and maybe your way, which is possibly a faster wrong way.