First, I would get the motor and drive working with an analogue signal into the drive, use a battery box or pot across a 9v battery etc.
Use 120deg commutation and with power on the hall sensors but the motor power disconnected, turn the armature by hand in both directions and the drive LED should stay green, if it goes to red, then the commutation sequence is wrong.
Once the LED stays green, then connect the motor power and try varying the speed in both directions, if it stutters or takes off or behaves oddly in one or both directions, then you have to try the six different armature connection commutations until the motor is controlled properly, once it is OK then you can connect the feedback loop.
Then the condition may occur where the encoder phasing is wrong.
i.e. the commanded direction contradicts the encoder direction.
Then you can either change the A & B pulse if single ended, or the A & /A if differential.
BTW, the hall spacing depends on the pole count, this will vary the amount of electrical revolutions per mechanical revs.
Al.
CNC, Mechatronics Integration and Custom Machine Design
“Logic will get you from A to B. Imagination will take you everywhere.”
Albert E.