Never had to kick out a scorekeeper. I have, however, worked the table a few times with other people who I thought should have been relieved.
I once removed a guy from the chain crew in football (ended up earning a 15-yarder and a trip to the parking lot) and did have my infamous varsity coach ejection where the guy came back claiming to be "game management."
Fun times . . .