IMO, counterbores suck. The few times I've used them on manual machines, they seemed to require a bit of man-handling, like to the point of being a bit rammy with them. The requirement seems to be high feedrate to induce lots of cutting pressure.
The gentleman's method of doing a counterbore is to interpolate a circle with the cnc.
Brass can be difficult to work because it wants to hog onto positive rake tools (like drill type counterbores). And a dwell mark on brass serves to harden that spot locally. The harder spot then requires that you boost the feed pressure to break through, which inevitably happens, then the tool gets a good bite immediately into the soft material beneath, and bam, it hogs in and rips the part out of the vise
First you get good, then you get fast. Then grouchiness sets in.
(Note: The opinions expressed in this post are my own and are not necessarily those of CNCzone and its management)