I try my best not to concern myself with which teams I'm assigned to officiate. Our association has an assignor that schedules all of our games. I've had the same Varsity Girls team four times this year. They are one of the top teams in the city. When I go there I try to stay as "even keel" as possible. Of course they remember me, that's human nature. My role is to stay professional and, along with my partner, give then the best game we can.
As far as the comments that were made to you and your pard, forget about it. If the same team with the same coaches and fans would have won those games, you guys would have been praised. Simply work a good game for them and don't be concerned with everything else. The coach who thought you were his "buddy" got a wake-up call.
In the future, I'm sure he will be more careful.