It sounds like a rigidity problem!
Try putting as large a diameter pin in the spindle as you can, and let it stick out as long as at can. Then place an indicator up close to the spindle nose, and set it to zero. Then with your hand try to move or bend the pin! This will show any looseness in the machine, and help you adjust / tighten it up!
Also, a cheap or reground end mill can also produce a similar finish, as not all flutes are sharpened equally or concentric.
Hope this helps!
Widgit
www.widgitmaster.com
It's not what you take away, it's what you are left with that counts!