[just another ref]

At a local jr high, the girls team is coached by a husband and wife team. The wife is a teacher at the school and her husband is a volunteer, but he does most of the coaching. I have known these people as friendly acquaintances for several years, our kids have played ball together, but that was about the size of it. Last week early in a game, I hear the guy say "You can't call the game this way just because you hate me." I thought I had heard wrong, but he said it again. During the second quarter, I was at the table and he was six feet away, and he said pretty much the same thing. I was stunned, and said the first thing that came to mind. "______, if you wouldn't talk so much, maybe we'd all like you better."
In retrospect, not the most professional choice of words, but I was trying to keep it light and at the same time tell him I had heard enough. He continued to mumble about the calling throughout the game: "...no, he's not gonna call that for us...."etc. After the game he came to me out on the court, I was going to stay away from him. "It's not fair to the kids for you to call this way just because you hate me." I was somewhat annoyed by now, but thought I used some restraint and said, "You flatter yourself if you think I called anything because of YOU. If you feel this way maybe you should stay home next time and everybody would be happier."

Am I waaaay out of line? How would others have handled this situation?