What should have happened here?
 

This was in a high school game in Illinois. On fourth down, the opposition was punting, but there was a fumble because the punter has butter fingers. The punter recovered and tried to run ahead. He was tackled out of bounds, and the play is blown dead.

One of the home team's players committed a late hit out-of-bounds after the whistle.

However, the opposition's illegal motion penalty was waved off and the opposition was awarded possession and got an additional 15 yards and a first down.

Correct me if I am wrong, but the home team should have received possession and lost 15 yards from the spot, right? Because the play was blown dead, there should have been a turnover.

You're making this play more confusing by using home and opposition instead of A and B or K and R, but it appears that you have the situation correct.

Without looking in rule book, I think the home team(R) would have to decline the illegal motion penalty to get the ball. Then it's a dead ball foul which would be 15 yards marked off against them and then 1st and 10 for R.

Illegal motion is not a dead ball penalty. So no the play should have not been blown dead for that penalty.

I have no idea who is the kicking team or who is the receiving team, so I have not idea how you enforce the penalties. You need to clarify that before anyone can really answer.

Peace

I figured home team and opposition was pretty clear. Let's try again.

This was in a high school game in Illinois. On fourth down, Team A was punting, but there was a fumble because Team A's punter has butter fingers. Team A's punter recovered and tried to run ahead. He was tackled out of bounds by Team B, and the play is blown dead. Team B did not get a first down.

One of Team B's players committed a late hit out-of-bounds after the whistle.

However, the Team A's illegal motion penalty was waved off and Team A was awarded possession and got an additional 15 yards and a first down.

Correct me if I am wrong, but Team B should have received possession and lost 15 yards from the spot, right? Because the play was blown dead, there should have been a turnover on downs.

Without trying to sort through your post:

When K runs OOB with the ball on 4th down, their series is over.

It's now R's ball. Any foul that occurs now is assessed and R's gets the ball, 1st down.

I do not care who the home team is, it only matters who is the kicking team and the receiving team as if you want to give the ball to someone. ;)

As said if the line to gain is not achieved, the dead ball foul will not change that status. And if the illegal formation was accepted then you would repeat the down, but still apply the dead ball foul.

Peace

You asked what should have happened?
 

Well,

Live ball foul for illegal motion against A.

Dead ball foul for personal foul againt B.

Play ended short of line to gain.

Team B has the choice of accepting or declining the penalty for A's foul.
Take the penalty it will 5 yards agaisnt A and then 15 yards against B, if the distance is enough for a first down, then first down for A, if not then repeat 4th down.
Decline the penalty and it will be B's fall 15 yards behind were the play ended for the personal foul.

Right, though this should be K and R. ;)

If the officials presented the options to B correctly, it should have gone something like this: "You can accept the penalty for IM and give them a first down after we enforce the penalty for the late hit, or you can decline their penalty, keep the ball, and start 1st and 10 from 15 yards back."

R probably kept the ball by declining the penalty. If that's what happened, it was not "waved off" (as the OP suggests).

We had a play on Friday night where a dead ball USC foul changed the decision of team B.

3rd down and about 10. There's a run up the middle that left A about a yard short of the LTG. When A's captain asked who the foul was on, I told him and the offending player flipped and drew 2 flags (one from me and one from the U). One foul, of course.

So, without the USC it would've been 4th and 1 and I'm pretty sure B would've accepted the penalty as it would've made it about 3rd and 20. But with the USC, they were able to decline the penalty and make it 4th and 16. Had A reached the LTG on the play, B would've had to accept both penalties to keep it 3rd and long.

All pretty straight-forward penalty enforcement (once we verified that A was short of the LTG on the 3rd down play) and it seemed pretty easy when I presented those options to the coach, but without having some simple principles down pat, it would've been more confusing.

There being no kick, there was no K or R. It was clear from the original post which team was A because the live ball foul was for illegal motion.

Only if that distance was sufficient for first down.

Probably. Possibly the original poster didn't see the option being presented to B's captain.

I think I've got it now except Team B does not have possesion.

Beyond that you have it correct.