This guy probably teaches the local “Baiting 101” referee class.

I have only threw out one fan ever. I was doing a 8th grade youth league game on a Saturday afternoon. It is my second game of four and have a young official for the first 2 and someone different for the last two. Its one of those set ups where the fans are behind the goals. Partner calls a foul and as I am walking to take the ball out I hear a fan in the stands yelling you guys are horrible, I look up and I kid you not he is in referee gear. I ask if he is working and he says I am working the next two after this game. I am red hot right now because I expect it from fans not other officials. I say you need to knock it off, as I am walking away he says loud then call the game right. I turn and eject him from the game. The supervisor takes him away as he is laughing and saying this is a joke. Game is over and here he comes walking up saying going to be hard to eject your partner. I call my league assignor and tell him what happened and he is furious, ten seconds after I got off the phone with him my partners phone rings and he says not a problem I can stay. Ten seconds later the supervisors phone rings and he says not a problem. Tells the guy he was ejected and the ejection last all day so he cant work. If the guy was just a fan I ignore it but the fact he would sit there in a referee uniform knowing how hard this game can be and talk trash was my breaking point.

Wow, that's an all-timer. You absolutely did the right thing, and good on the assignor too. That kind of nonsense would get you before the association disciplinary board for unethical behavior here...

Out, Damn'd Spot! Out, I Say! ...
 

(A Shakespeare reference (The Scottish Play) on the Forum, is that cool, or what?)

I would never eject a fan, that is, I would never take it upon myself to tell a fan that they have to leave. I could make such a decision, but I'm not interacting (and certainly not confronting) with the fan in any way (verbally, physically, pointing, gestures, etc.)

What I would do, and have done, is to instruct someone in charge (principal, athletic director, site manager, coach, janitor, even the police officer in the corner), to have the fan removed, and that I wouldn't restart the game until my instructions were fulfilled.

No way am I physically removing the fan, in fact, no way am I physically removing anybody, coach, player, etc. (but I will tell team members that they are ejected and possibly throw in the non-official baseball "heave ho" signal to let everyone know what's going on).

I will wait, even if I have to wait in the locker room, until that fan is gone, and I can be very, very, patient.

Note: After a few fan confrontations in parking lots after games last season (one that required a 911 call and police intervention), officials here in my little corner of Connecticut have been strongly encouraged to leave the game site together.