Had a similar problem years ago on a cnc where the encoder was slightly loose. Not sure what the mount is on yours, but it would hunt for position. The more it vibrated the more it would vibrate the loose encoder and this would make it worse.
John