Are you doing this on a drill press or a mill. Sometimes to minimize the burr it is best to use an endmill to drill the hole but this is not possible on a drill press.
An alternate approach which can minimize the burr is to have a mandrel that the aluminum tube slides over. There has to be a hole in the mandrel, of course, for the drill to enter when it passes through the wall of the tube. The burr gets pushed down into this hole and mostly gets pinched off by the drill as a it goes through the tube and into the mandrel; then when the tube is pulled off the mandrel most of the remaining burr is sliced off.
An open mind is a virtue...so long as all the common sense has not leaked out.