Here's my suggestion, and how I would handled from the start.
At the pregame conference, I would have stated to both "balking" coaches that if they did not give me an official lineup, then they were both waiving all lineup violations; be it intentional or unintentional, I would not rule on any batting out of order or substitution violations. The only official lineup is the one in the PU's possession, so if no lineup to check, there are no violations.
Had they proceeded as they did, my decision to the defensive coach is that he waived the right to make that appeal, so there is and will be no violation for me to rule. So how do feel about not giving me an official lineup now, coach?
Protest? Sure. Ruling: You waived your right to appeal when you didn't give the PU a lineup. Your statement that is an illegal sub cannot be verified.