I generally look if the tackler is playing the ball or the opponent. If there is a clean tackle and the other player falls over the ball (even if it's a hard fall) it's a no-call in my book.
When a defender lowers a shoulder and contacts the ball, then levels the opponent, that probably warrants a foul.
Severity of the contact is not always a determining factor however.

If the defender is playing the ball and misses and subsequently trips or runs into the other player, I call the foul.