Does your area have any kind of 'game management' present? Around here, schools are supposed to have some kind of personnel present to help out in situations like this. Here, I would go to the coach and them to get that person(s), and have them that those people need to follow the NYS Sportsmanship Rule - which includes spectators as well as players and coaches. If they persist, we then can have the game management to remove those people.
I actually had something like that twice this last volleyball season. Was working junior high level ball, when the visiting coaches came to me during the warm ups, and said a group of boys were making remarks to her players. Since we were in warm ups, I went right to the local game management - one an asst. principal, one an AD, and things got settled REAL quick.
The point is that it is not your job to enforce things like that, but the host school. If they do NOT provide that proper and sporting setting, then as an official have the ultimate option of not playing if you think it is not conducive for fair play. But it will never go that far - hopefully, especially not in this game.