We do it all the time with ER16, ER25 and ER32 holders.
There are two things to watch for:
Sometimes the ID tool holders are not perfectly one center; this does not matter as much with a boring bare but with a drill it is more important so you should check alignment.
Sometimes with a drill you can slowly push it back into the collet; we rig up a stop behind large drills so this does not happen.
One thing we have found handy is with these collet holders you can use corner radius endmills as small boring bars for shallow holes. Only one corner is used for boring and sometimes it possible to use the opposite corner for OD turning and save a tool change. You do have to reverse the spindle direction.
An open mind is a virtue...so long as all the common sense has not leaked out.