I know I read a post about this no too long ago, but it might have been on the softball board.

So here is what happnes. R3 and 1 out, infield playing in on the grass. B1 hits one hop to F4 who throws to F2. R3 is called out at home. But the BR thinks that was the third out. BR runs through first base, then he turns and runs back towards the third base dugout. The dugout was pretty straight behind the foul line, so the BR ran back to the dugout in the baseline. The defense threw the ball to F3, who was on the base. Then he went and tagged the BR, who by then had realized his mistake and was running back towards first.

Now I know 1st base is not a force out, so if the BR retreats back past first, does the defense have to tag him out, or do they just have to throw the ball to F3 and have him tag the base? Does it make a difference if the BR steps on first on his way back to the dugout?

By your description, the BR (obviously) abandoned his efforts to run the bases by heading to the (3rd base) dugout. If the umpire so judges abandonment, runner may be declared OUT. 7.08a

BR was tagged, so it is a moot point. However, if umpire does not judge abandonment, BR must be tagged, since he has already touched 1B.

In the instance where BR misses 1B (overruns or overslides),and he does not immediately return to the base (and the umpire has not determined abandonment), either the base or the runner may be taqgged for the out. 7.08j

I'd hold off on the abandonment call until the runner actually entered the dugout. Then there is no question. I've seen some of these idiotic AAU teams pull stunts (legal ones) to confuse everyone.