Even if you tossed the wrong fan, the point to "all" the fans was made. I wouldn't worry about it.
Around here, while game management is responsible for enforcing ejections, it is expected that an ejection will only happen if the official "pulls the trigger". Its unfortunate that the "culture" around here is as it is. I would dearly love to have a more aggressive game management. But with routine crowds of 800-1000 people and game management generally sitting in the corner waiting for us to bekon them, its how it works.
In my 7 years of officiating, I have ejected 5-6 high school fans (one this year). All but one have been fans of the same school. The other one was a GV parent who ran on the floor
during the JV game 
to protest an out of bounds call. He missed watching his daughter play her final varsity game because of the ejection as she became academically ineligible before the next game.
CYO games (Catholic Yelling Organization) are a trip and I have removed countless parents over the years. There really is no game management around, so we are on our own.