FriendOfGhost,
It's going to be a little hard to give you any exact tooling as it will depend on the job size and other factors. I can give you a few pointers though.
I have attached pictures of our fishtail and ball-nose end mills. I hope this will help with understanding how they work. Basically the fishtail will cut a square bottom leaving steps when 3d carving. The ball nose on the other hand will leave a little crater. This will allow you (with the proper code) to cut detailed images without steps (or at lease less visible ones).
One thing you can use fishtails for is rough out. Basically you use a cheaper tool (like a fishtail end mill) and cut the bulk of you material out. Then you use a ball-nose to clean up the last of the material. This is often faster and more cost effective.
Wood selection is something hard to direct you on. I am assuming that when you say "soft wood becomes like cotton" you are referring to the material fraying or leaving other artifacts behind in the cut. This is probably due to either the wrong tooling or the wrong feed. If you have the right tool at the proper feed and speeds then you should have no problems cutting almost any material.
If there is something I can help with please let me know.
John Torrez
Think & Tinker / PreciseBits