Keep spectators out of the game. If you try to eject him, you validate his argument. Unless he's inciting players or Manager/Coach, keep him on the other side of that fence.
If he DOES incite, call time, players in dugout, make the Manager[s] or TD/League Official deal with him. Maintain time on the field until he's gone.
Beyone ignoring him, tune him out.
