According to FED, when does the defensive team need to make an appeal on an illegal batter? Only rule I can find is 7-4-1h, which states batter is out when illegal sub is discovered.

For example, say a DH has used his re-entry and then late in game comes back to plate. Does appeal have to be made before the first pitch to batter? After first pitch to batter? After batter completes time at bat?

I found a reference in case book 3.3.1 AA, that umpire calls the illegal sub out once he is aware of the infraction. So, if original DH has used up re-entry and then becomes illegal sub, I can call him out when he enters box, after first pitch, after reaching base, etc? As long as it is brought to my attention before he legally touches home?

You are mixing up two rules here, Improper Batter, which becomes a proper batter if not discovered before the next pitch, See Rule 7-1-2b, and a "illegal player", which is addressed inRule 3-1-1 Penalty on pg 27. When timing has no bearing on when the player is discovered.

ok, improper batter deals with batting out of order correct? 7 hole (Davis) bats for 6 hole (Jones), Davis reaches base, 6 hole (Jones)comes to plate, defensive team appeals before first pitch to Jones. Jones is declared out and Davis is removed from first and is the new batter.

My DH scenario is: DH bats in first, reaches base, PR (not defensive player) come in to run. Third inning DH comes to plate (he has in effect re-entered). He gets on base and PR comes in. At this point original DH is done. This second PR is now to DH of record. Jump to the 5 inning when the original DH comes back to bat. Now, I know he is illegal. Do I call him out when he takes his place in the battter's box?

Well...
 

You could call him out, or you could simply tell the coach he's illegal when he comes to sub him in.

If the coach doesn't sub him in, since in FED the PU is responsible for keeping up with subs, I would just go to him and ask.

Coach you want to put in Jones but he's not legal.

I guess some umpires would go ahead and wait and call him out, but I like to be proactive and take care of these things before they happen.

Thanks
David

Yes- time has no bearing on when the illegal sub is declared out.

However, he has to be detected before the next pitch to the next batter for his actions to be nullified like advancing the runners and such (i think- not 100% sure on this one).

An ump should restrict an illegal sub, but not always call an out. But an illegal sub who was restricted and reenters is out and ejected whenever discovered, as long as still on the field.