A. you are not a mind reader
B. you do not have a time machine or a replay machine
C. you can only call what you see
D. you cannot see everything
E. regardless of the reason for the punch, you must eject
F. who cares, it's only a game anyway
If you want to view this as a learning experience, ask yourself if you and your partner used proper positioning and mechanics during the entire sequence. If you did not, and that contributed to not seeing the alleged first incident, then discuss it and figure out how not to do that in the future. However, if you did nothing wrong in that respect, then that's just the nature of a game that has 10 players spread out over the court and only 2 (in your case) officials who are expected to have wide-range vision.