If you intend to what amounts to step drilling the plasma cut hole, it is likely better to bore it than to drill it. A masonary drill can be sharpened so that it will drill hardened material from solid quite well, but it needs to drill the complete hole in one go because the drill point gets stabilized at the center of the hole.
When you try to skim out an existing rough, crooked, uneven hole, the masonary drill will not likely be stiff enough to drill in true position, and the vibration will be extreme and immediately knock the corners off the carbide.
If you began with mild steel, it is likely that the hard skin is very thin. You could grind that out of the hole with a carbide burr or small stone in a die grinder. Then you may be able to drill it with normal HSS drills.
I'd still advise boring the hole though, because I've seen some fairly awful holes cut with a plasma, but I'm think of thicker material that was difficult to pierce. A boring head in a mill will do the job. Interpolating the hole on a cnc is another quick way to fix the plasma cut hole.
First you get good, then you get fast. Then grouchiness sets in.
(Note: The opinions expressed in this post are my own and are not necessarily those of CNCzone and its management)