Am I right to say that the base umpire never calls the infield fly but points to the location of the ball.

It is the primary call of the plate umpire but the base umpire can call it if they feel they need to. The signal is a closed fist not pointing to where the ball is.

Raise right arm with fist closed. Verbally call, "Infield Fly."

That's our line up here anyway. And same with Ed, PU's primary call, but either ump can call it.

talk about it in pregame to get that kind of stuff sorted out.

ISF
The plate umpire shall determine and declare whether a fly ball is an infield fly .
To help the BU holds their hand in the air but doesnt call it .