JAR,
You need to add three more items to your list to get the full affects of worst case scenario reffing:
4. Your daughter is playing in the game
5. Most of the players are your students that you are currently teaching
6. Your wife is one of the coaches!
See you had it pretty good after all!
I feel for you though, it helps me to get past the annoying comments by thinking that I'm here for the kids, and like you said, "Tomorrow is another day!"
I've had fair success in shutting up the loud mouth parent(s), by asking them days later when I see them up town or at the next game (before it starts) if they would mind helping out with reffing future games, I say this sincerely and mention that we could use some help at the games, especially with reffing. Most of the time this shuts them up at future games and in a couple of cases they take the bait and try to ref a game with me, funny how it only takes one game of trying to reff these lower level games and the parent never steps forward again, they usually tell me at the end of their one game that they never had a clue how difficult it is to see plays, fouls, violations ect, half can't even remember how to blow the whistle!