Penalty Question

[Rick KY]

I had the following play, and made the ruling in a JV game Saturday...

4th & 21 at B25, A1 pitched the ball backward to A2 rolling left. A2 stopped, turned and threw a legal forward pass (incomplete) about 20 yards beyond the NZ. B1 was rushing A2, the passer, with hands high as he let go of the pass. B1's hands then came down and grasped the face mask of A2 as he dragged him to the ground without letting go of the mask. The contact was late, after I said "Ball's Away!"

Originally I dropped my flag thinking face mask (15 yards), and gave the preliminary signals for PF Facemask. As I consulted the A captain it suddenly occurred to me that this was the passer, so I changed my ruling to roughing the passer, thus carrying an automatic first down for A. The PF Face Mask would not get A another series.

Was I correct in changing the foul called in this case?