Roughing the kicker

[DeltaRef]

During a scrimmage kick, with A lined up in proper kick formation, B lineman comes in totally untouched.

B is charging hard and commits to a tackle attempt against player A in the kicker position. B actually leaves his feet.

Somehow, just before impact by B, A kicks the ball cleanly down field.

Offensive coach wants roughing the kicker, as the kicker is hit hard and driven to the ground.

As ref, I declined to throw the flag, because defense has committed to his charge.

I reasoned that this would be similar to tackling a quarterback as he makes a pass - not roughing.

Was I correct? NFHS rules.