I will (attempt to) explain a fault transition which in normal operation is an impossible signal sequence with quadrature square wave.
As you have already discovered there are to signals lets call them A and B being approximate square waves 90 degrees out of phase.
Following just assumes forward is as described, but that could be reverse in fact. If you were to swap A and B then the following logic will magically come good.
Line A goes high/low in a square wave manner, one for each line of encoder (in most peoples opinion, but depends on who wants big numbers on their spec.) and so does Line B
While rotating in FORWARD:
1. Line B low to high transitions occur when A is high.
2. Line A high to low transitions occur when B is high.
3. Line B high to low transitions occur when A is low.
4. Line A low to high transitions occur when B is low.
While rotating in reverse:
5. Line B low to high transitions occur when A is low.
6. Line A high to low transitions occur when B is low.
7. Line B high to low transitions occur when A is high.
8. Line A low to high transitions occur when B is high.
When rotating forward the sequence is always 1-2-3-4-5-6-7-8-1-2..
When rotating in reverse the sequence is always 8-7-6-5-4-3-2-1-8-7..
A direction change can occur at any sequence position like 1-2-3-4-3-2-1..
A valid stationary position can be 4-5-4-5 or 8-1-8-1.
In a stationary position ONLY ONE channel can 'rattle' at any one time.
Both signals NEVER CHANGE at the same time. This is a fault transition.
Depending on how circuits are implemented a clock transition can be on every edge of both signals or only clocked by A sampling B.
When an edge on one line is detected the state on the other line is sampled. The sampled state depends on direction.
Of the 8 states, you can only move from one to the next or previous.
Drawing a timing/transition graph with the 2 channels soon gives some idea of the possible and the impossible following the rule 'both signals NEVER CHANGE at the same time'.
It is easier to experiment with this than describe it ! Please verify my logic.
Can someone will explain in 1 line what I should have written.