The material you used was almost certainly some alloy, pure aluminum is very rare because it is so soft.
All alloys need to be age hardened, also called tempering, to attain the best physical properties and machinability.
You could try doing your own age hardening; Heat the ingot to 75 - 80% of the melting point and quench it as rapidly as possible in a tank of cold water. This fully anneals it and gives it close to the same condition throughout.
For age hardening you can simply wait a long, long time like several decades and it will age harden at room temperature; but probably you are in a hurry. So you can age harden at an elevated temperature.
It is possible to do it in a normal domestic oven, particularly the self cleaning type that has the ability to get extra hot. Simple bake it at a temperature as high as the oven will go for several hours. It can take something like ten or more hours but if age hardening is overdone the material can become brittle.
One approach you could try is to anneal the ingot then cut it into several pieces and age harden these for different lengths of time and see if the machiniability differs.
Do not quench it after age hardening, let it cool down slowly.
You may not find any difference in the machinability after this because some alloys do not harden as much as others.
An open mind is a virtue...so long as all the common sense has not leaked out.