Batting Glove Worn By Pitcher

[Fritz]

Often see coaches remind their pitchers to take off a batting glove (worn inside of their fielding glove of course) before starting to pitch. But for the first time, had a coach come out and argue that the opposing pitcher needed to take his batting glove off (for the record, the batting glove was essentially black, and his fielding glove was all black). The pitcher complied without my partner or I saying anything, but it got me looking into the rule book for a reference.

I couldn't find anything and wondered if this is another myth along the lines of "the hands are part of the bat."

Am I overlooking something? Is it different between OBR, NFHS or USSSA?