This is a great topic. I deal with this all the time because of the small area I live in. The schools around here for JV, Freshmen, and Middle school games always use refs from the town and alot of times they are teachers at the home school. There are only of few of the these guys that I think should not work at there home schools because of giving advantages, but for the most part I think that the games go well and it saves the schools a lot of money because they don't have to pay milage. I work games at the school I gratuated from (3 years ago) I now live about 40 miles away. I do have some fans think that they should get the advantage because they are at home (the "homer" advantage as they tell me). Usally their comments just give me a good chuckle because they have never even opened a rule book and have no idea what they are talking about. I have worked JV games there for 3 years now and both coaches have told me that they are happy when they see my name for their game. The other 2 schools I do JV games at are the town I live in (which I don't know alot of people because I haven't been here that long so I don't have alot of people say anything to me because they think that I am an out of towner) and a town 10 miles south (which was the high school that my high school combines with for the football season so I know a lot of people from that town.)
Short story: The first varsity baseball game I ever did was one year out of high school and it was with my "old" high school team. I made sure that the AD of the home team that hired me knew that and he told me that he had seen me call games and said he confident that I could call a good game. Before the game it was kinda uncomfortable because I was tring to be professional but it was hard because everyone knew me. First inning I called a kid out tring to steal second which took them out of the inning and my old coach got a little upset but I stood my ground and told him what I saw (it was an easy call the kid never mad it to the base, ball and glove was between) The rest of the game went well as it turned in to a blow out and there were no real big calls)
In the area I live in for all of the under class games I work for basketball I have to get my own games so it becomes alot easier to get games at schools were you know the AD and also that are close to your home as schools here in the northwoods tend to get spread out. One of the conferences that I get varsity games for I have to travel 2-3 hours to get to the schools but that is were I could break in because they are in need of officials.
So I guess what I am saying is that I would not give up a game because of the town it was in or who was playing if they feel comfortable giving me the game than I will take it. I have reffed both of my younger sisters and my younger brother in their games. My sisters were in a summer high school tourney that I reffed and My brother is in 7th grade and I have done many of his games.
Just call the game regardless of who is playing and call it like you don't know anyone and don't take the things people say personally, just like the others said they are just yelling at the stripped shirt it doesn't matter if it is you or someone else, they will yell. That is until we make all fans pass a rules test before they are allowed in to the game!!!! (like that will happen

)