Hurray, I finally succeeded in making my part after getting a proper machining vise and lots of trial and error:
2mm, 4900RPM, 0.5mm depth, 200mm/s: tool snaps on 1st outer perimeter
2mm, 4900RPM, 0.5mm depth, 120mm/s: snap on 2nd or 3rd layer when starting to clear pocket from inside. 4 flutes tips were all severely chipped.
2mm, 8700RPM, 0.5mm depth, 120mm/s: lasts longer, but still SNAP ! only 1 flute tip was chipped
4mm, 6800RPM, 0.5mm depth, 120mm/min - success!
as before, tool tip distance to spindle brace was minimized to 4.5cm.
Note: I have no way of measuring actual spindle RPM, so I estimate by (VFD frequency / 400) * 24000 RPM / 1.1
lessons learned:
1. using a tool with <= 2mm diameter is almost certain to break for steel, no matter how fast it's spin
2. need to use high RPMs to prevent tool from chipping or breaking - can't rely on those figures like 100 feet/min for steel that's intended for much stiffer machines
3. don't always use carbide if there's noticeable vibration? I wasted a lot of carbide bits. Maybe I would've been better off with HSS or cobalt steel?
Other notes:
The most vibration happened when the Y axis (with most load) was the furthest away from an end - about 1/4 of the way. So I should've cut with Y closer to the ends or reducing gantry height might also help
liquid cooling unecessary? I tried sulfurized thread cutting oil when experimenting with 2mm bits. It reduced vibration, but I don't know if the thermal shock contributed to the flute tip chipping. I didn't use any cooling for 4mm and it turned out OK.