Sticking around is definitely a plus. A seat near the scorer's table or a spot "away" from a majority of the fans is one I would choose. Assess the situation and adjust your remarks/ response accordingly as JRutledge noted. One smart a** remark I use sometimes is "Thanks, all of my friends tell me that" or woudl even ask "Which rule did I miss of 1 thru 10 and ask them to help me understand it"

A couple of years ago, I filled in and worked a doubleheader because of a last minute illness to one of our JV guys and had a motormouth on the front row tell me toward the end of the JV game: "You'll never be a varsity official." Laughed and said, "You're probably right." Came back out to do the varsity a few minutes later and said, "I just got promoted. Sorry." Motormouth's facial expression...priceless.

Moral of the story? Don't sweat the idiots. Do what's comfortable to you. Ignore him, say thank you sarcastically, or come up with something witty. Nothing will ever change those clowns anyway.