Without question I think you have to take into consideration ALL of the factors surrounding a game. That is why we are referees. Anyone can read and memorize the rules, game management and judgement set the great apart from the good.
PS...Interesting that you bring this up, Esse Baharmast recently quoted in an interview commenting on this.:
http://www.soccer365.com/US_home/Fea..._38_4473.shtml