ejection

[newump]

18+ over summer league. mostly college and ex-college players.
i'm BU, close play at first, BR ruled safe. F4 complains briefly about the call, F1 then responds to F4. "don't worry about it, he (meaning me) is an *******" I was about 20 ft from F1 at that point - about the same distance as F4. the comment was not directed at me, but he was referring to me and i clearly heard it. there is no history between me and this player or team. it was the first game i'd ever worked in that league. i immediately ejected F1.

I went back to the the A position, which happens to be right next to a large contingent of fans of F1's team. one fan in particular starts to get on me about the call and then about the ejection. he repeatedly demanded to know why i ejected the player. i refused to respond. some of the opposing players told him to shut up, etc. this went on for several innings. he went on and on about how i ejected his player for no reason, blah, blah, blah. i con'td to ignore him and eventually he slowed up a bit. his team was getting crushed 12-0, so that also led to his frustration.
he didn't curse or get too personal, the worst he did was call me a clown and said i don't deserve to get paid.

should i have responded at all to the fan?
should i have demanded that the coach control his fans?
should my partner have said something to the fan or coach?
thoughts?? how much is too much from a fan?
thanks.