Illegal Substitution?

[Viking32]

Time out on the field. A has authorized conference near their sidelines. Warning is given to coaches that the time out is almost over. Team breaks authorized conference and runs directly to the LOS. On their way there, the ready for play is signaled. As they line up, they realize they have 12 on the field. There is a little confusion as to who should leave, so A calls time out. Is this illegal substitution?

They did not "break the huddle" with 12 (it was an authorized conference, not a huddle). I treated it as if A12 ran on the field, realized he shouldn't be there, and started to run off before the time out was called. There was not a snap imminent. Any thoughts? Which rules should be applied in enforcing this one way or the other?

Thanks