It's been long enough since I've worked a game that I can't honestly remember when I shake my partner's hand, but I suspect it was as I left the field. It certainly never caused an issue.
I think the concern is overblown. I referee soccer. In soccer when the match is done, the three referees all congregate in the center circle and shake hands there before heading off the field. No one ever says anything about this because it's done that way all the time (and, let's not flatter ourselves, no one cares much how the officials leave the field, so long as they leave).
|