[TimTaylor]

1.Signal you describe is "illegal hit". It's mostly used for prolonged contact, but also for illegal hit during a serve in a couple of situations, including the one you describe.
2. Rule 8-1-1: A serve is contact with the ball to initiate play. The server shall hit
the ball with one hand, fist or arm while the ball is held, or after it is released by
the server.
3. Sequence of cards in volleyball is yellow, then red, and finally both yellow and red together. Yellow is a warning for minor violations. The coach loses right to stand and must sit on bench, but there is no other penalty. Red card , penalty point is awarded to opponent. Combined yellow/red is penalty point to opponent plus disqualification - similar to getting a 2nd direct "T" in basketball. The official can also skip the yellow and go straight to red or yellow/red if the nature of the conduct warants it.

From your description of the situation, the first card (Y) was likely for unsporting conduct. 2nd card(R) was probably for continuing to argue or a comment made after first card was issued. From the penalty section of rule 12:
1. Unsporting conduct shall be penalized as follows:
a. Warning (yellow card) for first minor offenses;
b. Unsporting penalty (red card) for first serious offense or second
minor offense and loss of rally/point awarded to opponent;
c. Disqualification (yellow and red card) for first flagrant offense, second
serious offense; or third minor offense and loss of rally/point awarded
to opponent. Offender is disqualified from further participation in
match and may be removed from the vicinity of the team bench.

Hope this helps answer your questions.....