Hi powerfade,
I sent you a pm but promised I would post here as well as it might help others.
I think part of both your original and current problem is the amount of over hang you have on your z axis. Do you have a way to measure that from the center of your y , or x, depending on which you have your z mounted? I have a 5X10 r/p machine, and I have vibration as well, but my machine is much more massive than yours.....so I don't really think it is the r/p as much as the overhang.....and I have that too.
Mike
No greater love can a man have than this, that he give his life for a friend.