Yeah - I wouldn't do the game.
I don't have any kids so I don't have to worry about conflicts but I'm surprised your assignor or group don't ask you in advance for any potential conflicts.
In fact, our group recently instituted a policy of "15 years" post graduation or else you can't work a school's games. So technically, even though I have no connection to my old high school besides being a graduate and a fan of their basketball teams, I have been told I can't work any games associated with the school until next year.
