4 players on the court?
 

What's the call on a team having only 4 players on the court because a player decided to leave? This was a substitution for all 5 players on the court. Then one of the players who who just came in decided to go back to the bench after the ball was put back into play. This left the team with only 4 players on the court. He can not come back onto the court until a dead ball, otherwise it's a technical, correct?

Incorrect....when there is confusion over a substitution, the player that should have been in the game is allowed to return at any time as long as they do no do so in a way that gives the team an unfair advantage.

Where is this located in rule or case book?

From the Interps
 

Following a (a) charged time-out; or (b) a lengthy substitution process involving multiple substitutions for both teams, A5 goes to the bench and remains there mistakenly believing he/she has been replaced by a substitute. The ball is put in play even though Team A has only four players on the court. Team A is bringing the ball into A's frontcourt when the coach of Team A realizes they have only four players. The coach yells for A5 to return, and he/she sprints onto the court and catches up with play. RULING: In (a), the officials shall stop play and assess a team technical foul for not having all players return to the court at approximately the same time after a time-out. The technical foul counts toward the team-foul count. In (b), the officials may permit play to continue without penalty. A5's return to the court was not deceitful, nor did it provide A5 an unfair positioning advantage on the court. COMMENT: Even though neither situation provided A5 or Team A with an advantage, teams are expected to return to the court at approximately the same time following a time-out. The officials should have also followed the prescribed mechanics and counted the number of players on the court, ensuring each team has the legal number of players. (10-1-9; 10-3-3)

All that has been posted so far is correct, assuming there was confusion.

However, perhaps the OP implied that it was clear the player was not confused and rather just said to himself, "f%*^ it, I'm done with this," and goes back to the bench. I'd have to be 100% sure of this, but if I was, I'd come out with a "leaving the court for an unauthorized reason" violation. Then during the dead ball we can address making sure the team has five players on the court if five team members are available.

Case Book 10.3.2 Sit B

If this were the case, I would think that it would be leaving the court for an unauthorized reason to demonstrate resentment, disgust or intimidation. I think that is a T, not simply a violation.

What rule citing would we use for the NCAA games?

Only if such attitude is directed towards an opponent or the officials. If he's just disgusted with himself, or the coach, or he's just ready to be done with the game....
Frankly, a disqualified player with an attitude is (almost) authorized to leave the floor in my game. I'm not going to make him stand there while his coach figures out who is going to take his place.

Could I? Sure, but I don't want to be the guy who clears the lane for the FTs so we can penalize the violation after the FTs. I'm certainly not calling this T unless he clearly directs his wrath to us or his opponents.