Just a couple of silly questions here? -
If the contact was enough to cause the player to go OOB - would that not be sufficient enough to be a foul?
or
If the contact was sufficient enough to cause the player to lose control of the ball OOB would that not be a foul?
This contact seems suffucient enough to have affected the play - so if you choose to give the ball back to the team responsible for the ball being OOB (last touch) then you are setting aside a rule.
I understand where the other comments are coming from light contact is not always called - however we rule it incedental because it has no real effect on the play this does not seem to be the case here.