I had a game with one of those guys once. At his whistle I froze watching. I didn't really look at him closely - just enough to know his fist was up and he was trying to communicate for me to report the foul. When I saw everything was calm I moved to endline and stood next to him. He asked if I reported the foul - I said "what foul?"
I thought he was going to kill me.
It was the longest game of my life.