Pitcher runner contact?

[frozenrope22]

Happened this weekend. 14U NFHS rules. No runners and no outs. Batter hits a high pop up between home and first base. Runner is running right down the chalk. Pitcher comes over to catch the ball. Pitcher is looking up and runner is focused on getting down the line. There is incidental contact between pitcher and runner on the foul line. The contact knocks the pitchers glove off. Now the pitcher attempts to catch the ball without his glove but misses. Runner reaches first and the ball falls about 6 inches foul. Ball doesn't touch anyone and clearly lands foul but only by a few inches.
Umpires confer and call the ball foul and return the runner back to hit. The contact was truly incidental and didn't appear intentional on either player but I felt the pitcher would have easily caught the ball if he had his glove on.
I thought the runner had to avoid the first fielder making a play. While the ball was in the air there was no way of knowing it was going to land foul.
Was this the right call?