[johnnyg08]

Here are some things that might be of interest. This is a good question and you forced me to find rule support for what I thought was correct.

From the BRD:

OBR:When a coach or manager confers with a defensive player,a trip is charged only when – before a pitch or intervening play – the player goes to the mound or the pitcher comes to the player. (8.06 CMT)

Rule 8.06 Comment: If the manager or coach goes to the catcher or infielder and that player then goes to the mound or the pitcher comes to him at his position before there is an intervening play (a pitch or other play) that will be the same as the manager or coach going to the mound.
Any attempt to evade or circumvent this rule by the manager or coach going to the catcher or an infielder and then that player going to the mound to confer with the pitcher shall constitute a trip to the mound.
If the coach goes to the mound and removes a pitcher and then the manager goes to the mound to talk with the new pitcher, that will constitute one trip to that new pitcher that inning.
In a case where a manager has made his first trip to the mound and then returns the second time to the mound in the same inning with the same pitcher in the game and the same batter at bat, after being warned by the umpire that he cannot return to the mound, the manager shall be removed from the game and the pitcher required to pitch to the batter until he is retired or gets on base. After the batter is retired, or becomes a base runner, then this pitcher must be removed from the game. The manager should be notified that his pitcher will be removed from the game after he pitches to one hitter, so he can have a substitute pitcher warmed up.
The substitute pitcher will be allowed eight preparatory pitches or more if in the umpire’s judgment circumstances justify.