My sympathies, Kong. I haven't found a way to make a finish cut in aluminum to break chip dependably, that's why I mill it as much as possible, particularly when working inside of a hole.
Are you roughing with this same tool, all the way to finish? What are you starting from, a drilled hole? How many cuts is this insert taking before the final finish? I'm just wondering how perfect the edge is before the final cut. If it is suffering built up edge, that can skuff up your surface, too.
The volume of metal you are removing still generates a fairly stiff chip. I would try a finish cut of .005" depth, in an attempt to keep the swarf building very slowly in volume, and being very lightweight. You might have to retract from the hole at a suitable position (say as the tool moves over a corner or something), to remove the swarfnest before continuing. Hopefully, you can then move right back in where you left without leaving a toolmark.
First you get good, then you get fast. Then grouchiness sets in.
(Note: The opinions expressed in this post are my own and are not necessarily those of CNCzone and its management)