Been there...
Girls varsity game earlier this year. Two excellent teams. Team A pressing and B1 passes to B2. B2 catches the ball and gets bumped by A2. I pass on the call because it's unintentional, fairly mild contact and I didn't think it would affect the play. It turned out to be a bigger bump than I thought because B2 lost her balance a bit and after a second or two traveled. I then called the late foul. Coach A was
not happy (A fans weren't real thrilled either). I happened to be in front of the A coach so I gave a real quick explanantion (not time for a whole dissertation on holding the whistle). I've reffed A enough that the coach let it go, but holding your whistle isn't always an exact science.
In that case, I was going to cause unhappiness whichever way I went. As bad as it felt at the time, I still think it was the right call.
Z