Should I have been the hard-***?

[Ref_ Fred]

Not your place to chew him out, but it is your place to tell the assignor of this particular event not to ever team you with person again. That would probably a better way to get your message across.

Asking to not officiate with him again, is just pushing the problem on someone else and not dealing with him. remember when on the court you are a team. If he screws up, you both screw up. So deal with it right then and there. He'll see how much of an unprofessional person he is. Although you have to do it with professionalism.