Umpire wants yellow card, Referee refuses - OK?

[officialtony]

Jr. Hi Match. I am referee and have an umpire assisting. Team A completes a rally point scored and Team B coach jumps up and loudly says " how about a carry on two of those hits ? ". I respond by saying they were legal hits in my opinion. I sense no problem here. As she sits down she says " yeah right ". Umpire pulls his yellow card and wants to give her unsportsmanlike card. I disagree. I felt she said what she wanted to and sat down. It was neither violent nor disrespectful ( in my opinioin ), and I considered the fact they were losing their second game by a margin of more than 18 points. I considered it more frustration than unsporting.
I denied the yellow card and my umpire came over to confer. I again denied the yellow and he reluctlantly put it away.
Was I wrong? Do I back up this kind of call when I truly believe there was no unsporting behavior? Do I HAVE to back up his call?