I checked the rule book, which clearly lays the responsibility on the scorekeeper to sound the horn when an illegal server contacts the ball (5-5-3-e). The referee also has responsibility to determine the alignment of players of the serving team (5-3-3-c-7) and has general supervision of all officials (5-3-34-a-1). You could argue that if the scorekeeper misses it and the referee misses it, the umpire could still blow the whistle upon the improper server's contact, because under Rule 5-4-3-a he or she would be "assist[ing] the referee," but would be a stretch to say the situation was "clearly out of the referee's view." . . . out of his mind, maybe, but not out of his view.
Thus, I come back to jumping on the sound advice that Homer and Andy have given of letting the point play out (even if you are 100% sure of the violation) and during the next dead ball checking with the scorekeeper. If you were right, you notify the referee of the error and correct the score appropriately.
Nick