Steppers and servos both are capable of 0.001“ and way finer but first off, you're trading speed for resolution in some cases (the pitch of your ballscrews). Second, that won't be the problem in my opinion. It'll be backlash and slop and flex in the entire system which probably can't total more than like 0.0005 for you to hit your goal. So not only do the bearings and ball nuts and mounts all have to have less than that amount of backlash, but your system, across the largest spanned member, has to also flex less than that under load.
I'm taking some liberties in the math here but just trying to make the point that the motors are probably the easiest part to get "accurate".
That said, you're making this because you want to and many people have made great machines on their own! Just be realistic about what it'll do.

Sent from my SM-G960U using Tapatalk