When I do get a chance to see a V Game or arrive early for a JV game, I try to sit as far away from the others in the stands as possible to avoid these issues.
Once, I even heard a, "Shh, watch what you say. He's an official." Like I would do or say something?
I suppose many of us have heard similar to you.