There is a lot to be learned by working this level of ball. There's patience. There's how to deal with hotheaded has-beens. There's how to deal with clueless and abrasive never-weres.
The purpose of most church leagues is not competitive sports. The purpose is fellowship, exercise, and sportsmanship. Don't allow the idiots (and there are always at least a few) to derail the purposes of the game. Don't take crap. Don't lose any sleep over calls you made, didn't make, missed, etc.
Do be professional. Do act the part. Do not be surprised at how many people are completely clueless about what your part is.
Above all, communicate your expectations about player behavior in appropriate ways. Then back up those expectations.
Seriously, the worst that can happen is that you get invited back.
(No, that wasn't a typo).