Team K in scrimmage kick (punt) formation. 12th man of K (K58) runs onto field thinking he's 11th man. He got to the middle of the field and was waived back, so he ran back off of the field. I was HL so I threw my flag for Illegal Substitution. My R waived it off and said there was nothing wrong with the play.

The reason I threw it is because from the rules, a substitute who comes on the field and becomes a player must participate in the down. However, after reading the rules book it doesn't appear that K58 ever became a player since he never a)lined up in the formation or b)gave an audible signal for someone to come out.

I guess my question is was this Illegal Substitution or not?? (I am not sure it was now)

A substitute doesn't become a player until he substitutes for someone else. Sounds like you have no foul here.

I agree, no foul...simply a substitution that was not completed, therefore the replacement may leave with no penalty

Would things change if this "sub" communicated with another player or an official and was still waved off?

No. A player is not substituted for until he leaves.

We're asking for trouble if we start throwing flags when high school kids run on when they're not supposed to. The team that wants this penalty probably will reconsider within that same quarter when they do it.

Yes, especially when what they did was not illegal. :)