Catcher's Obstruction on backswing

[Bluefoot]

Men's ASA SP Rec league. Last night I called Catcher's Obstruction on when the catcher stuck his glove forward and hit the batters bat on the backswing, BEFORE the forward swing, as the pitcher was pitching. Some members of the defense went nuts and could not understand how I could make that call, since it was on the backswing. Today, I can't find anything in the rule book that states a difference between forward and back swing. I just told the team that the catcher cannot interfere with any part of the batter's swing that prevents the batter from trying to hit the ball. It's the catcher's fault for being too close or reaching too far forward. And the batter certainly did not intentionally try to hit the catcher's glove. Was this ruling correct? Thanks.