Yes, the shank of the endmill will rub unless you can grind it slightly undersize between where the flutes and and the chucking surface of the shank begins. If you have no facility to do this, you can carefully grind this 'neck' with an angle grinder or something, while the tool is running slowly in the spindle. Move the tool out of the holder a little so you don't mark the holder. Clean up the dust afterwards, and clean the toolholder too.
You don't have to radically undersize the tool, a couple of thousandths will suffice.
First you get good, then you get fast. Then grouchiness sets in.
(Note: The opinions expressed in this post are my own and are not necessarily those of CNCzone and its management)