When you're profile cutting there are three things that are really, really important. They are RIGIDITY, RIGIDITY AND RIGIDITY, OH, and did I mention RIGIDITY.
It looks to me that with that Kress spindle you ain't got it.
I use cutters as small as 1/16 diameter in my PCNC 1100, and have never had a finish like that. I typically run 4,000 to 4,500 RPM 12 to 15 IPM feed. If I want a really good finish on the sides, I will slow the RPM down to 3,000 and leave the feed at no less than 12 IPM.
When I have large amounts of material to remove, I'll use a larger end mill and leave .005 to .010 stock for finish.
You can buy GOOD PARTS or you can buy CHEAP PARTS, but you can't buy GOOD CHEAP PARTS.