Which are you doing; flycutting, which uses a single point tool very often HSS, or face milling, which use a head will multiple inserts very often carbide?
If you are face milling with carbide that can give you a poor finish on Delrin. Most carbide inserts do not have a really sharp edge; you can see it easily with a powerful magnifying glass, the edge has a minuscule radius. This means the Delrin gets rubbed instead of getting cut so it tends to roughen up.
If you are flycutting with a HSS tool you need to sharpen it with zero top rake and have a really fine sharp edge. You also need to make sure the corner is not sharp but is ground to a bit of a radius so the edge in contact with the Delrin is wider than the distance you feed per revolution.
Also you need to be sure your head is correctly trammed. Even a small misalignment over a 3" diameter will leave noticeable semicircular grooves in the face.
An open mind is a virtue...so long as all the common sense has not leaked out.