Slight Body Contact on drive to basket

[buckeye38]

New official here. I am having a tough time, especially as lead in 2 man crew, judging if slight contact is a foul when the defender is step for step with dribbler on a drive to the basket. I do not want to punish the defense for keeping good position and not pushing but I seem to get in trouble with my no calls because the game becomes more physical. There is no hacking or pushing just what I consider incidental contact. Also players know to fall away from the shot making it look like a push or has happened. I can't seem to get consistent with this call. Has anyone had to overcome this dilemma?