ASA - Infield Fly Rule

[Chess Ref]

ASA 14U. R1 on 2b, R2 on 1B. I'm PU. Batter hits this little strange bloop over the Pitcher's head. Pitcher made no effort, ball lands about 4 feet in front of 2B. All runners advance one base. No one said anything. BU calls "Time".
Comes jogging over to me and tells me he had called "IFF". Nobody,including myself heard him-so I guess that part is irrelevant-but when he told me he called it, I replied "Thats not your call-it's my call". I got the I've been doing baseball for a gazillion years spiel and either of us can make the call.
Since nobody heard him, there really was nothing to "fix", so after a discussion he went back to his position and away we went.

My question I have is IF someone had heard him, am I obligated to accept his calling of the IFF, or could I have overruled him and placed the runners where I believed they would have ended up?