For what happened in your specific situation a player technical foul is required. You did darn well for not knowing the exact rule.
Now if the fifth player had not returned to the floor, I would have allowed play to continue. There is now no penalty in the book for this (since the rule change for leaving the floor). I do wish that the NFHS would issue a clarification or case book play on the situation in which the player does not return.
Lastly, if you can stop the ball from becoming live on the throw-in, I highly recommend doing it. Even if you aren't sure that the team only has four. If you just suspect that there is a problem, it is better to try take care of it during a dead ball and possibly be wrong, than to let the play happen and end up having to call a T that you could have prevented. What's an extra few seconds?