Drills and end-mills both have advantages and disadvantages for this application. Unfortunately neither is probably going to give you what you want .
Drills might have the depth of cut you need but as you stated will want to pop into the previously drilled hold. Trying to mill with a drill is also a problem since they are very weak with a side load put on them. Also be aware that a normal style drill is only the spec'ed size at the tip, the rest of the length is slightly smaller.
End-mills will cut the material and irregular shapes fine but you're asking for a tool with a 20 to 1 aspect ratio (20x the diameter of the cut). The highest standard tool we make is 8 to 1. Even if we made a custom tool you would have a very hard time using it. Any runout in your spindle or inconsistency in your material would break the tool.
John Torrez
Think & Tinker / PreciseBits