I noticed a team doing this
 

Everytime a sub entered the game they were given a rolled up towel by the asst. coach and gave it to the player who was to come out of the game, and when they got back to the bench they gave the towel back to the asst. coach. I assume this was their way of making sure they knew the player subbed for came out of the game. If they sent in 3 subs they had 3 towels etc.

Twelve Men On The Field ???
 

Isn't this kind of like the football rule?

That's My Story, And I'm Sticking To It ...
 

Who's responsibility is it to see that only five players are participating from each team?

By rule? The team.
By mechanics? The officials.
Anybody else? Head coaches, assistant coaches, players, substitutes, and the table crew.

If all of the above do their job correctly, you'll never have six players participating at the same time. If one of the above screws up, the "buck stops" at the team, and the team is penalized with a technical foul.