Balk call last night

[Justme561]

I am seeking opinions regarding a balk call I made last night.

OBR - Runners on 1st & 3rd.

The pitcher comes set. The batter asks for “time” but I say “no, stay in the box”. As the pitcher begins his delivery the batter steps out of the box. The pitcher holds up and does not throw the ball. I call “balk”.

After I explain to the defensive coach that I had not called “Time” everything was fine.

After the game one of my partners said that I should not have called a balk in this situation. He said that the actions of the batter caused the balk and I should not have “rewarded” his actions. I was taught that it is the pitcher's job to throw the ball regardless of what the batter does.

Was I correct?