I always tell the assignor up front I will not ref any league games from my son's division. However, similar to others I have reffed my son's rec games ONLY when pressed into action due to other ref's not showing or only having a very junior ref for the game.
I always talk to the coaches, who all know me and say - your call, I can keep the books and clock as always and you can go with one ref or get another parent. Since I have reffed for many yrs and am one of the more senior officials "at rec level" they usually let me call the game. Never had any complaints - I'm probably fortunate.
Funny aside - last year, I T'd up my son for unsportsmanlike conduct in one of his games I reffed. He fell on a slippery floor and slammed the ball down. WHACK! He knows better. One of only 3 - T's I gave all yr.
Mom was none too happy with that one!