Actually, your question is a very good one and is indicative of a higher level of officiating a game.
You are correct that by the book this action is a foul. However, you are also correct that at that level of play there is a great deal of physical contact that is accepted by the players and allowed by the referee.
The referee has to read the players and use good discretion in deciding whether or not to deem such actions to be fouls.
Nice observation.
