Infield Fly Rule

[rbmartin]

This happened to me yesterday. FED Rules. Game 2 of a double header bottom of last inning with score tied (not that any of this matters).
I am HP. Bases loaded with 1 out. Potential winning run on 3rd. Before batter gets in box, I give infield fly signal to my partner and he acknowledges it back to me. Batter hits very high popup to right side. From my vantage point I am not sure it's going to be an easy catch and am not sure how deep it will travel (it was very windy) so I refrain from declaring an "Infield fly - batters out!" If my partner at the "C" position had declared it I wouldn't have argued with him but I chose not to and neither did he. Ball was caught grass in short right field by the 2nd baseman. After the play I have to admit it did end up looking like a pretty easy play. Anyway, no runners attempted to advance. No problem. The ball was deep enough that if the catch had not been made there would have been zero chance of a double play .
Now for some (unknown to me) reason the defensive coach starts yelling at me asking me why I hadn't called the batter out via the infield fly rule. I siad "it's a judgement call and I chose not to."
My question is, in deciding to declare an infield fly or not, should I consider the location of the ball and if there is a possibility of a double play or is my only consideration whether the ball "can be caught with ordinary effort"? Isn't the pourpose of the rule all about protecting the runner?
Anyway the shortstop booted the next play and home team won so life goes on.