When to step in?

[hooper]

Here's a situation I saw recently in a junior high game:

Team A's coach and fans are riding the refs throughout the first half. In the second half, the ref addresses the fans about their behavior. Moments later, ref calls a foul on Team B and Team A's coach says "That's it. Fouled out!" The scorer's table says no, one more foul before fouling out. (Scorer's table is run by parents and volunteers from home team, which is Team B.) Coach approaches scorer's table and begins to argue. Fan (and relative of scorekeeper) from Team B comes to aid of scorekeepers. Team A's coach gets up into Team B's fan's face and says he'll see him in the parking lot after the game. Refs ignore and continue game.

What would you have done?