You can actually use it to your advantage. It'll just give the coaches more reason to pick on you, and more reason for you to throw them out.
(I worked with a partner once who always wore glasses while officiating just so the coaches had something to make fun of him with. He wore contacts everywhere else.
)
Of of the greatest referees I've ever heard talk about the philosophy of refereeing and basketball was probably 5'6" at the most. Don't let it get to you. Enjoy it.