When is a conference charged?

[skinner51]

A game I was at, the defense called time, came out and made a pitching change. No charged conference for changing the pitcher. The coach then walked over to the dugout, then turned and went out to second base to confer with his infielders. As the coach crossed back into foul territory, the umpire informed the coach that because he had crossed the foul line in going to the dugout after the pitching change, then turned and went back out to the field to talk with his defense, he would be charged with a conference. The coach replied that it was a dead ball situation while the pitcher was warming up and that he is allowed to come back onto the field during a dead ball situation.

The umpire stood by his guns and charged the coach with a conference. Was that the right call??