Should I Have Run Him?

[YooperRef]

I had a JVG game last night where the following occurred in the second half. I am lead administering the throw-in. Visiting team is inbounding the ball on the endline near their basket. V1 inbounds the ball, steps on the court then steps back OOB and runs 15-20 ft. along the endline (OOB the entire time) and re-enters the court on the opposite side of the lane. Surprised me a little but I whistle the violation for unauthorized leaving the court. V coach runs down to the endline line where I am, grabs my arm, and asks what call I've got. I explain that leaving the court in the manner his player did is a violation. His response, "The hell it is" (loud enough for players and probably fans to hear). I T'd him up. Report the T to the table and let him know he's lost the use of coaching box.

I'm thinking that I should have ejected him for grabbing my arm (not roughly) or possibly issued a T for coming on to the court and a T for his response. It all happened pretty fast. Any thoughts?