That is very common using a die because it is tricky getting centralized forces on the handles of the die, especially if the part is held in a lathe and you are leaning over trying to hold it.
If you try the lazy man's trick of resting one arm of the die holder on the saddle and turning the chuck it will almost certainly run off.
The most precise approach is to thread cut them; what type of lathe are you using CNC or manual?
If you cannot thread cut another approach is to turn a pilot on the end of the threaded section down to the ID of the die. This acts as a guide to start the die on center and parallel and you then machine it off after the thread is done.
And practise your technique for balanced rotation of the die.
An open mind is a virtue...so long as all the common sense has not leaked out.