I had that problem with one set of encoders that I had (not knowing they were differential) and the problem was due to the fact they drew 120 ma.
Geckos are limited to 50 ma draw on the encoders. Mine worked and then faulted out intermitently like you indicate. As soon as I changed encoders, everything was fine (or you can use an external power supply for the encoder-see documentation for proper hookup)
As a side note, I have also run 1000 line encoders with absolutely no problem from that aspect. Mariss has given out some information on how high you can go before expecting problems and in my configurations, the limit has always been in the 3000 line area YMMV
Art
AKA Country Bubba (Older Than Dirt)