Did I Jump the Gun?

[tmp44]

15-16 yr olds, FED rules, I'm BU. Team B (visitors) is up 7-2 going into the Bottom of the 5th. Team A has made a comeback and it's now 7-6 with 1 out. Team B brings in a new F1 who on first pitch throws a ball. On the pitch, I was in the C. As the pitcher gets the ball back from the catcher, he looks into the Team A dugout (3rd base line) and yells "Oh so we're going to start talking sh!t now!" I immediately called time and ejected F1. At no time did I hear anything come from Team A's dugout, and there had been no problems before this in the game. Team B HC didn't complain to me directly, but I heard him say later that it was one of the weakest ejections he had ever seen. Did I jump the gun here? If so, what should I have done? For you coaches out there, what would your perspective of this had been if you were 1) Team B coach and 2) Team A coach?