It is a factor. I would not say it is "the" factor. But it certainly is a factor. And anyone says you call fouls without knowing something about the result of the contact (which does not have necessarily have to do with the ball going in the basket is lying or does not have very good judgment IMO). And yes it usually is a bigger factor on marginal contact.
Peace