Call it on the sound???

I had a situation in a summer league I'd like to throw out there for some comments. Late in the game, team A up by a few points, obvious fouling situation. On a missed free throw by A, A1 gets rebound, I am screened by other rebounders, my partner evidently did not see B1 reach in, but I hear the slap on A1's arm/hand, which could have been committed by anyone on team B. Is it right to blow the whistle and call the foul, or do we really need to see it? I did blow the whistle and just assessed the foul to the team B player of my choice. Any thoughts???