George,
If you are using High speed steel tools (such as drills), you need to limit the largest circumferential speed of the tool to 100 (or less) feet traveled per minute. For a 1/4" diameter tool, this would be about 1500 rpm. The forumula is:
1200/(Pi * tool diameter).
For carbide tools you can run 3 or 4 times faster.
A variable speed drill would put you in a much better speed range. You might want to use a 3/8 or 1/2 rated drill, in order to get sufficient gearing to do something. Mind you, none of those tools have spindles or chucks that are made for the side loads of milling, but you can certainly drill with them.
First you get good, then you get fast. Then grouchiness sets in.
(Note: The opinions expressed in this post are my own and are not necessarily those of CNCzone and its management)