Infield fly mistake??

[DaveASA/FED]

Ok, the post below got me thinking, I had the following situation in a game.

R1 on 2nd, R2 on 1st 1 out. Left handed batter hits a fly ball toward 1st base as PU I call out "Infield fly, batter out if fair." Ball is dropped foul, I call "Foul Ball." A left handed batter come up and hits a grounder to get safe at first. Defense coach comes out and appeals BOO. Turns out B3 went to dugout after my infield fly call, she nor any of her coaches heard my "if fair", nor did they question the fact it was a foul ball. B4 who was also a left handed batter (these are the only 2 on the team) came up and started batting since B3 went to the bench. As this girl is the same build and has the same color hair I never noticed that is wasn't B3, and since left handed batters are not that common I never thought to question it. Now the question is what should have been ruled?? I ruled B3 out for BOO, and returned runners to bases and put B4 back up to plate. But I now question if this was the right call. I am thinking about the actions of an umpire causing the player to be at risk. I know that they should have known the situation 18U rec ball(most 13-15 year olds), but wondered what the experts have to say.