So I changed the steppers to closed loop hybrids, put in limit and home switches, and changed to a computer water cooling system. I also got an AXBB-E controller with UCCNC. All that seems to work just fine so far.
The parts I kept - the spindle and nowforever VFD, aren't working anymore. I did change the VFD-to-spindle cable to a shielded one. I added the 4th pin (ground) to the spindle and soldered it to the spindle top cap inside. The cable ground is then also attached to earth which is also attached to the VFD ground. Originally the VFD was never earthed.
The controller sends a 0-10v signal to the AIN1 pin on the VFD. at S24000 it's 9.87v, at S120000, it's 4.97v and so on. The X1 pin is receiving 13v since it's a different common on the controller.
I see the freq on the VFD display go from 0 to a max of 379.something when I change the S values, but nothing happens with the spindle. I did remove the resistor the VFD originally had between the 12v pin and the AIN1. Putting it back doesn't seem to help.
All the settings on the VFD are what it used to be on the old setup, which used to display a constant 400 no matter what. Not sure why that's different now.
I'm looking for any advice on how to troubleshoot this. I'm a total newb.
Here are some pics of the new setup (I still need to clean the wires up a bit)