After game situation

[youngref33]

I was wondering if I handled this situation correctly?

Working an AAU tournament.
One team is more aggressive and reaching all the time.
Parents and coaching are saying I am being bias to one team.
I my opinion I was not.

After the game..two parents preceded to come up to my partner and myself I complain about me being bias. My partner supporting me the whole time, one of the parents turned to me and said "you know what you should do?"
I turned to him and said "sir, I really don't care what your opinion is"
Then he proceeds to tell me that I would care if "we took this outside". I looked to him and said "real mature sir" and walked away.

Talking to another official, he told me I was the professional and handled it wrong.
Any suggestions on how to handle dumb situations like this?