I don't think you should throw him off but definitely chew him out.
We've all seen those little huddles between referees in the NFL, no reason the three of you can't discuss it while the BJ keeps everyone away and U watches the ball/spot. Right? Probaly a good thing to pregame. Defintely chew him out though
Not worth throwing him off your crew unless he has had other problems.
That's me though.