If you are reaming in mild steel, brass, or aluminum you can fairly easily take out several thou with the reamer so you can even go 1/64" undersize for a 1/2" reamer. Typically you would go something like 2% of the reamer diameter undersize for the drill between the limits of never having less than about 0.005" or more than about 0.025" to remove with the reamer.
For your application you may be better off doing the reaming in the drill press. It is very hard to run a reamer in by hand without creating a bell-mouthed hole; reamers are difficult to hold parallel with the hole by hand.
Clamp your work onto the drill press table for drilling and leave it there for reaming. You may need to position the table as low as possible while still getting the drill through the work so that you have clearance to get the drill out of the chuck and the reamer in. Do not use power to drive the reamer. Turn it by hand either by turning the top pulley on the spindle or putting a small rod in the holes in the chuck to act as a lever (unplug the drill for both of these). Feed very gently while turning the reamer and use a bit of libricant such as tapping fluid. NEVER turn the reamer backwards: If you find it jams pull it straight up out of the hole and start again feeding even more gently.
An open mind is a virtue...so long as all the common sense has not leaked out.