I did what you have asked about. I started when I was 14 working our local church leagues. I got my drivers license on the day of my first "real" scheduled softball game. I worked HS baseball while still attending HS (the only restriction was I couldn't work games in my school's region). I started working city league basketball. I even got my first varsity football assignment (but sadly was in a car wreck on the way there). Then at 19 I took a couple of years off and came back. Then packed it in entirely once I got into my career. Now, at age 36, I'm back. I WISH I HAD NOT PACKED IT IN ALL THOSE YEARS AGO.
There will always be things that coaches will dislike about officials: height, weight, age, gender, attitude, reputation, style, mannerisms, etc. etc. etc. Fortunately youthfulness will fix itself. Hang in there. As near as I can tell, the problem isn't that they think you're too young to be a good official...the problem is that they see your age and assume that you're not mature/professional/experienced/etc. enough to do a good job. That's a perception you CAN overcome.