Infield fly rule after the fact

[prekowski]

Situation was runner on 1st and 2nd and one out. Batter hits routine popup to 2nd base and runners at first and 2nd hold close to their base, batter barely moves from batters box. Umpire fails to make infield fly rule call, 2nd baseman drops the ball and picks it up and gets force at 2nd and then throws to first to complete the double play. When other team voices protest that IFF should have been called, the umpire declares that's right, it should have been called and therefore I am calling it now, so only batter is out, double play is erased, and runners are allowed to return to their bases they held before the batted ball. When defensive team argues to the ump that he did not call IFF during the play, ump responds with yes, but that he is calling it now (on dead ball).
Protest was lodged immediately and recorded in scorebook at time. Who has an opinion on whether the protest should be upheld.