If you have a uniform filler ie just one material of a single size, the possibility of uniform packing rather than random packing is high. This will allow crack propagation very easily along the plane of the packing.
The whole point of a mixed size aggregate is to force any micro crack that is trying to grow through the material to keep changing direction.
Each time it does so, it loses energy, and eventualy stops, so no macro crack appears.
I'll try and find a book I have with a title something like "Why materials are strong", possibly by David Pye. Good explanations about engineering qualities of reinforced materials.
John