As has been mentioned, properly torquing the ER holders is key!
Another thought. The advantage of the collet chucks for finishing is less runout. Runout is a real issue as your cutter gets smaller. Think of the max allowable runout in terms of a % of the cutter's diameter. A 1/16" endmill can tolerate about 1/8 as much runnout as a 1/2" endmill.
In my shop, I only use collet chucks for cutters less than 1/2" in diameter for that reason.
FWIW, great deals on collet chucks available at Glacern and Maritool. Probably others too, but I have experience with those.
Cheers,
BW
Try G-Wizard Machinist's Calculator for free:
http://www.cnccookbook.com/CCGWizard.html