did i do the right thing?

[CajunNewBlue]

Well i've done it now.... told my boss "I aint doing it" and it all started with a phone call from a "mommy".
last thursday im working the Pony rule based recreation league in a nearby town and its a nice little 12u game. everything goes fine, i mean its not 12u gold teams but both teams work hard and game is uneventful. except ..... for this one daddy who keeps hollering from the backstop "you have to get out of the box"... over and over and over and over and over and over and over.... so i finally turn... put my fingers to my lips in a "shush" manner and turn back around. not a peep for the rest of the game. So anyways i get a call from my boss that his boss got a call from a mommy that i wasnt making the batters move from the batters box on a passed ball (and let me tell you there were about 100 passed balls in this game) and on 99% of them no play was involved. So i tell my boss "of course i didnt tell them to vacate the box" thinking he was throwing me a trick question. long story short.... we argued, I asked for a rule cite that says I have to move them or call them out for not moving (yes, i know to call them out if they interfere).
I looked and looked and cannot find a rule reference that says this is part of my job and i personally feel that by making them move and if they get hurt I am actually more liable.
Question is..... did i do the right thing?