Given that you have limited options you are going to have to compromise. The variable speed drill is probably the wrong choice, it does not have bearings suitable for side loading. The router can work but it is necessary to keep the cutter size down around 3/8" or smaller. Aluminum can be cut at a surface speed of several thousand feet per minute and if you do the math you will find at 26,000 rpm the sfm of a 3/8" cutter is around 2500.
Lubrication is essential, absolutely essential if you have a hodge podge alloy cast from all manner of stuff. These castings are going to have a very low temper and will be very soft and gummy for machining. The alloy will tend to stick to the cutting edge of the tool without an form of lubrication. If you do not have access to proper cutting fluids a perfectly good alternative is automatic transmission fluid just painted on the material along the line of the cut, or sprayed on from squeezy bottle.
Rigidity is likely to be a big problem. It is best to cut in climb milling mode which tends to pull the cutter up onto the work if the machine is not rigid and backlash free.
To improve the machinability of the castings you could try a crude tempering procedure, if they are small enough. Heat them to about 470F in a regular oven then quench them in water; this more or less anneals them. Actually you need to go higher than 470 but most ovens peak out around there. However, if you have access to one of the self cleaning type put the castings in and run the cleaning cycle which is quite a bit hotter than 470 and then quench them. After this annealing procedure you want to age harden, also called precipitation harden, them and this is done by holding them for several hours at about 350F, several being something like 12 hours. As soon as you try machining any castings that you have tried to temper using this procedure you will know if it has worked because the difference in machining is like night and day.
By the way, nothing wrong with carbide wood bits for machining aluminum they can work quite well. I use router bits for machining radii on aluminum parts and they last forever.
An open mind is a virtue...so long as all the common sense has not leaked out.