Since you directed the defense to remove the bat, it became their responsibility regarding any possible blocked ball (IMO).
I would point out, however, that if you were using ASA rules, the written ASA rule (7-1B) says the on-deck batter is to be in the circle nearest the offensive team's bench.
|