If the mill has a modest top speed of 4000 or 5000 rpm, you can probably run it full speed while cutting aluminum. But that is not saying that you should run it at full speed all the time
For hand feeding, you can only maintain limited control, normally, one tends to speed up and slow down with every turn of the crank. So a higher speed is probably better in order to even out the 'fast phase' of hand cranking
But, the highest speed is not necessarily good for the tool, if you are only creeping along with the feed. In other words, no use overspeeding if you are underfeeding.
Too much speed and too much feed will often plug the tool as the softened hot metal welds itself into the flutes of the endmill. If coolant is not possible, then slow down to prevent that from happening.
Plunging does not require a different spindle speed, but usually the feedrate will be maybe 1/4 of normal. This is because a 2 flute center drilling endmill does not have optimum center geometry to really permit good chip flow, in comparison to a drill. Because the endmill has sharp flutes with relief for cutting, and a flat end, it tends to walk and wobble around when plunging, and makes an oversize hole. The more you feed it pressure on the plunge, the more it walks.
We used to reckon on 400 surface feet per minute with HSS endmill doing heavy cuts in aluminum. With carbide, or with HSS in finishing cuts, its practically got no limit so far as the tool is concerned, so long as the coolant can get into the cut, and the chips keep coming out.
Surface feet is reckoned by the largest cutting circumference of the tool. Imagine the linear distance a point on that circumference would travel in one minute.
400 * 12 in/ft = 4800 inches per minute. Divide that by the tool circumference and you get spindle rpm.
First you get good, then you get fast. Then grouchiness sets in.
(Note: The opinions expressed in this post are my own and are not necessarily those of CNCzone and its management)