Actually you do need to tighten them quite hard; for an ER32 it is several tens of lb-ft of torque.
My approach is to tighten them as much as is physically possible using just my hands and the wrench that fits the collet nut, without hurting my hands too much.
Using just your hands, not a hulking great length of pipe on the wrench nor banging it with a hammer, just your hands, it is impossible for you to overtighten them; unless you are a real gorilla with enormous hands.
You may find however, that to loosen them you have to resort to a small length of pipe slipped over the wrench; they seem to 'settle in' during use due to heat and/or vibration.
An open mind is a virtue...so long as all the common sense has not leaked out.