Be aware that an encoder will only give you incremental position ie how far the motor has turned. So what you will get will be a pair of quadrature (A and B) signals that you will have to use to count up or down from a starting value. For absolute position you'd need something like a resolver.

The connector is more likely to be a 9 pin Dsub than a VGA connector. Take the encoder apart and trace the wires to the line driver to see how they are connected up. And expect to replicate the line driver receiver at the other end of the cable, to convert the signals back to TTL for the Arduino. It's conventional stuff but if I've lost you, you may be on a learning curve here...