When appeals can or cannot be made has no bearing on this. This is an umpire correcting his call. There are different "deadlines" (if you will) by which a correction must happen; after that, it is too late.
In the situation you presented, it was too late, since a pitch had been thrown.
I can see the umpire doing what this one did in an informally officiated game, because even though the next pitch has been thrown, no action has occurred that would make it impossible to correct the umpire's error. If I would have done something like this, however, (that is, do the wrong thing for the right reason), I would not have made it a complete "do-over." IOW, the result of the pitch would have stood. I'd have just sent the runner home. But, that is just an opinion of the best way to do it wrong!
