I've also found that its easier on you if with body language you convey that you are watching ... Probably every game you will get a complaint that someone left early, mostly done just to keep you as the umpire honest IMO .. but if you are obviously watching its just easier on you .. "How do you know she didnt leave early you werent watching" type convos or complaints.
Also .. if you do ring up someone.. then that keeps the team honest and its hard to deny you arent watching..
A bonus ring up is if you ring up someone on the complaining coaches team.. Ie coach team a) "blue you gotta watch, they are leaving early" .. next inning if you get to ring up one of his runners, you get bonus points...
Hey, you told me to watch and I am!