Custom holders created in Rhino and applied to madCAM is something that has been discussed for the future of madCAM 5. Currently, you are stuck with the cylindrical holders. Having said that, you could have a custom holder as part of your machine model, but it would be up to you to spot the collisions.
Dan
(Note: The opinions expressed in this post are my own and are not necessarily those of CNCzone and its management)