Thanks everybody for the responses. WOW....I thought our association was the only ones split on this issue. Actually, I believe the NFHS rule book covers this in Rule 3, Section 6, Article 2f: Action or inaction which prevents promptness in putting the ball in play is delay of game. This includes:
f. Any other conduct which unduly prolongs the game.

Of course, as a wing man, I would surely try my best to let Team B know they need to come out to the field or it will cost them at least 5 yards. Very grateful that I haven't had any team ignore me on this warning.