I worked a frosh/JV double header last night that I picked up, so I feel your pain.
All I can say is that I feel no obligation to respond to comments, if they cross the line deal with it immediately and
never warn twice.
Always remain calm and speak with confidence and be concise.
What I've found, and it helps me, is that most coaches and fans are clueless. Rarely does anything they say bother me, mostly it is a form of amusement. Plenty of times my partner and I have a good laugh during a game because of a coach.
Take it in stride, keep it in perspective, and if all that fails just whack the next one.