At what point do you engage/remove an obnoxious fan?

[rfp]

I make it a point to completely tune out fans, unless I happen to hear loud, abusive or foul language and know exactly who said it -- so I can get him/her ejected. But the other night, in a 30-point blowout, after calling an obvious foul on the losing team, a parent of the losing team in the stands nearest where I'm standing makes some comment about me stealing my game fee for my work in the game. Being down 30, it was a pretty ridiculous comment. I usually don't pay attention to any fan comments, but this one really irritated me, it was so stupid and uncalled for. Is there a point, shy of the obvious, that you'll remove a fan for a stupid comment like this? Or ignore except for swearing?