I've got a reindeer antler and it's hard as nails but I'm reluctant to cut it as someone dragged it all the way back from South Georgia for me.
I picked up an antler tang at a car booty, after pulling off the bottle opener, I pecked out a cup with that with no problems apart from a couple of chips to the tang point.
I'm as far as you could be from being an expert but I think the mallet may have some bearing, using a material that is softer than the tool may help to avoid stressing the tool.
I guess that, apart from experimentation, the only way to really get to grips with this and other rock art questions is the spade.