I always remind coaches that THEY have far more to do with what is in the rule book they play by than ANY umpire does!
It is true really. Coaches and leagues influence the content of rule books, NOT umpires!
Anyway, I agree that you open yourself up to liability if you allow a safety rule to be ignored, EVEN if it involves adults. You can't let it happen. If Babe Ruth baseball wants adults to wear a mask while warming up a pitcher on the field or in the bullpen, and that is in their rule book, that rule should be enforced!