[Nevadaref]

You are correct that the coach's comment was insulting and worthy of a T.
Others have told you that you should assess this immediately, unless the opposing team has a scoring play in progress, then wait as you don't want to unfairly deprive them of that.
Now please take this as kind advice and reflect upon it: you were greatly mistaken to ignore the coach during the first game. By the 2nd qtr of that contest you should have addressed his unsporting behavior. As our beloved JR used to say, nip it in the bud!
By ignoring it for an entire game, you sent the message that his behavior was permissible, and what you allow, you condone! Don't ever put yourself in this situation again. Take care of business by sending the message nice and early that you are an official who will be respected. Show the coach proper respect, as well, but you must be firm and establish that you won't be taking his garbage all game!
By the time that you decided to deal with this (even before game #2 was far too late as it would have been taken as you harboring ill will from game #1) the molehill had already become a mountain.