Apart from what the coach intended, what he did was submit an illegal lineup with only 8 batters listed, he listed one substitute, and had one rostered player he did not list on the lineup.
On that last point, it doesn't matter. She can still enter the game as a sub. (4-1-A-2)
If B3 is considered an unreported substitute for B4, then you have the problem of what to do with B4.
If B4 is considered an illegal re-entry for B5, then she is not batting out of order. So, she is an unreported illegal player, not BOO.
Batting out of order is an appeal that must come from the defense.
An unreported substitute is a protest that must come from the defense for there to be a penalty.
Since the defense appealed BOO, and since there is no BOO, the appeal is denied. Play ball and see if either coach can figure out what the infraction is and either correct it (offense) or file a proper protest (defense).