[Jim S]

Agree, you did right.
What we have here is an umpire confusing the passing game with the rest of the game.
We do give the R the calls on the QB for throwing illegal passes (dumping) and, in most cases, roughing.
But in your case the QB was just another runner. If you would have called it on the HB, then the call was correct on the QB. There might have been upto three flags on this play if the U had shifted back to the QB because he was approaching the LOS. The R's because he was taking the QB, yours because he was coming at you, and the U's because he was watching the QB as he approached the LOS.
Did you ask the U if he had seen the mask pull?