substitution or deception?

[ccRyder]

In last Friday's game, we had a "situation" and I thought we got it right, but wanted a second (or third) opinion.

Ball was ready for play and team A was in the huddle. They broke with 11 players. A1 came over to line up near me, the linesman and saw his substitute come in and line up in his spot, inside the numbers. He continued to run off the field. The substitute was inside the numbers - whether he communicated with the player who came off, I do not know.

The new wide out looked at me as most do to see where my feet were positioned to make sure he was on the line. He did not communicate verbally. We let the play go. The other team was now saying the substitute had to be in the huddle. The closest (I could find) NFHS case book play 3.7.5 says the substitute needs to be inside the numbers and not be deceiving. The other team obviously saw him as they were yelling "he has to be in the huddle"

Should we have flagged for substitution violation?