High helix angle flutes and high shear angle facemills seem to work the best for me. I also use flood coolant, as it is important to keep the cutter cold, so that the tendency of chips to weld to the tool are lessened. I think this accounts for the scuffing of an otherwise nice surface. Also, the chips need to be removed immediately from the cutting zone, so either flood or air is required to do this.
I would face mill a large surface, rather than zipping around it with a small tool.
I've found that Iscar helimills will do a really excellent job for finishing the bottoms of pockets. You need the special polished insert for aluminum, though. But for side milling, a high helix multiflute on an endmill designed for aluminum is going to give best chance at best results.
First you get good, then you get fast. Then grouchiness sets in.
(Note: The opinions expressed in this post are my own and are not necessarily those of CNCzone and its management)