Numbering exception

[Altor]

Then one of his teammates must be, because they only have 4 ineligible numbers on the line.

I think the question stems from the way the exception is written..."any A player numbered 1 to 49 or 80 to 99 may take the position of any A player numbered 50 to 79."

Expanding on the OP's scenario...
#81 has been the end, #55 the tackle. On 4th down, #31 comes in for #55. #81 takes the tackle position and #31 is on the end.

I think the OP is questioning that because #31 came in for #55, he should take the tackle position. I would point out to the OP that the rule only uses the word "player" and never uses the word "substitute." So, #81 would be using the exception in this case and would be perfectly correct in doing so.