Refs authority???

[crazy voyager]

What do you say about this?

Audience behaving badly. In a pause the away team asks me to send them away. I say
"I'm not responsible over the audince and have no authority over them, please ask the game administrator. He's responseble for the gym"
The team gets quite, upset, to say the least. And they try to dictate my authoroties claiming that a ref their last game threw audiance out. They didn't calm down until I said "sit down or I have to give you a technical". I did noting and I didn't think the audiance were bad either, I don't think the coach ever asked the administrator.
What do you say? Have I the power to throw audiance out? I've always been taught the referees powers end where the court does, coaches, players and officials. Not the audiance.