Mechanics Question

OK. Here's a mechanics question.

R1 on 2b. R2 on 1b. One out. B3 hits an infield fly. Kinda windy, but F4 only takes two steps and waits for fly ball. BU raises right arm with closed fist (infield fly rule signal). PU sees BU's raised arm, but elects not to invoke infield fly rule. Ball falls untouched. Now R1 advances to 3b, where she is ruled out by force (F4 retrieved ball and threw to F5, who was standing on 3b).

After wards, PU tells defensive coach that, in his judgement, F4 did not get to ball with little effort. Conditions were too windy (huh?).

Question: Who's call is infield fly? And what do you do as BU if, in your judgement, it is an infield fly and you have already given the signal (but not vocalized)?