If the contact is slight, and it's questionable whether it caused the fumble, I'm probably passing on the contact and giving B the ball. If the contact is slight, but clearly caused the fumble, I'm probably passing on the contact and giving the ball back to A. For this particular play, I'll referee it that way in the first quarter and in the fourth quarter.
I'm not likely to factor time and score into the decision. Nor am I likely to factor in the offender's star power. However, if the offender is a problem child...that's likely going to be considered. Heavily.