An aluminum-specific Carbide tooling is not worse than any HSS tool in surface finish.
With a proper tool i normally get mirror-like surface finish at about 0.0025-0.003 ipt chipload
Whatever marring you see is because of the dull tool.
These pockets were done at full 1.5" depth of cut with a .5" 2" long 3 flute Niagara Elite endmill. Even at 0.010" chipload and significant scalloping because of very hight feedrate, you can still see reflection of the tool.
For aluminum when tolerances do not exceed 0.002" i do not even bother to use a separate finishing tool.
Btw 4 flute endmills are most often designed for anything BUT aluminums. This means they have different rake angle that affects how the chip curls and needlessly increases cutting force, which in addition to higher number of flutes leads to a lot of chatter.
Sometimes even with very stubby cutters, tool with improper geometry "shivers" or causes more shearing than cutting and produces cluoudy finish.
Long and boring story short: Use tools designed for the purpose. Generic endmills that claim to work for all meterials tend to not be very good for anything.
http://zero-divide.net
FSWizard:Advanced Feeds and Speeds Calculator