Doing all that will take some time, but do it.
In the mean time, you're still the ref and still in charge. Don't go out there and act mousy. Act like you know what you're doing. Look like you know what you're doing. Talk like you know what you're doing. That doesn't mean be cocky or condescending, but exude confidence. Fake it if you have to, but act confident. It'll help a lot as to how your calls are received.
Welcome to the fraternity