hy gian
go with bearing nuts so to avoid useles torque on the collet ( that leads to micro-damage and less clamping force ), and be sure that the tool shank that goes inside the collet is blank ( thus not coated ), cilindrical ( no flats ), and mount it dry ( not only clean, but also dry), and use a H7 collet ( designed for a h7 clamping range )
17k rpm : is there a balancing machine ? and what is the maximum rpm of your spindle ? you know, so to avoid killing your spindle at max rpm, with an unbalanced toolholder
like other said, a toolholder with a smaller collet may help ... if it fails, and you wish to buy a hydra holder, it is no guarantee that it will work; consider shrink-fit ( definetly lighter than a hydra ) or weldon ( cheaper, possibly the screws will fly out, so craft another ones with small play, and adjust your cutting specs acordingly, because they deliver a higher tool run out, and that means that your final mould surface won't be as nice as when machining it with a tool with less run-out, but it may still be ok/salable )
or it may be that your tools are poor quality : if possible, change them often / kindly