Make sure you are using an aluminum alloy that is suitable for machining; 6061, 6262, 2011, 2024 are all okay.
Always use some coolant/cutting fluid; aluminum tends to stick to the tool and jam up in the flutes of both drills and mills.
For milling use two flute cutters to get best flute clearance for chips; spin the cutter very fast and feed gently. Just brushing the cutting fluid on ahead of the cutter will work or rig up a mister or sprayer.
For drilling also feed gently and 'peck'; that is go down about the size of the drill diameter then retract to clear chips and allow cutting fluid into the hole.
And don't get discouraged if you do still occasionally break drills and cutters; just remember what works and what doesn't work and eventually you will get things sorted out.
An open mind is a virtue...so long as all the common sense has not leaked out.