I've only had to remove 1 fan in HS ball and have threatened to have a few removed in AAU ball. Never any incidents with any of them after the game though.
Maybe b/c I'm 6'5" 260lbs or maybe b/c I have a such a friendly smile
but all the folks who have ever recognized me away from games have always been friendly.
Only "incident" I have ever really had with a fan was a few years ago in my 3rd or 4th season of officiating--a fan came on the court after a state level 15U AAU game and told his players "You can't win playing 5 against f**king 7!". I was already in a particularly bad mood that evening so as I walked off the court I looked directly in his face and said "f**k you!!!" For some reason that caused the fan to become very animated and that drew the attention of our very large security person. Turned out the guy had already been kicked off the premises earlier in the day for harrassing officials and had been seen drinking in the parking lot.
Now-a-days I would just ignore, or report, such an incident. Too many shootings going on around here to be confrontational.