More speed, go faster.
The rough dull finish you are seeing is the result of the material tearing and rewelding to the workpiece.
As the part dia gets smaller you need o spin it faster to keep the same surface footage. Think of it this way, if you put tiny wheels on your car the engine would have to spin much faster to move you down the road at 55 MPH. If you put 20 foot tall wheels on it the engine would push you to 55 MPH at idle.
I suspect you may have damaged the cutting edge or have built up material on the top face. Examine the cutting tip for chips, side wear, or buildup with a magnifying glass or microscope.
Yes the oil will help if the speed is too slow but once you weld some material to the top of the tool it's all over and the oil won't help anymore.
Material welding to the top of the tool is know as built up edge (BUE). This is caused by too low of a surface footage and is easily cured by cranking up the speed.
The only thing I use HSS for is bandsaw blades and drill bits, but then again I've got a building full of carbide so my tooling costs are pretty low.
Can you get someone to mill the holder down? If not how about grinding it down on a bench grinder?
Bob
You can always spot the pioneers -- They're the ones with the arrows in their backs.