Ger21- That epoxy will get harder over time, and after a few weeks will be harder than it is now. But it will never get that hard unless it has been through an elevated temp post cure process (heat to about 70'C for about 12 hours, then 45'C for a few days).
My CNC machine has 3 epoxy handwheels I cast to use on the stepper shafts. Two were post cured properly and are fairly hard and the 4mm grub screw has held fine. The third wheel was the last and (being in a rush) I didn't bother post curing it. The wheel has warped in shape from the tension of the grub screw and the 4mm grub screw stripped after a while (as the epoxy was lot softer). Now it has a 5mm grub screw and I treat it gently.
4mm is a very fine thread for semi-cured epoxy! I would be worried about the threads failing. The brass inserts might be a good choice, or maybe those 4-spiked steel nuts they use in wooden speaker cabinets that go in the other side?