OBS and missed base

[WhoMe?]

ASA slow pitch AA Rec. league. The batter hits a line drive to Left field and is on his way to second were F6 has blocked the bag with his leg while waiting for the ball. Runner slides into his leg..F6 catches the ball and slap tags runner and manages to sling the ball into right field.
Up pops the runner and he heads to third without ever touching second base and makes it without a throw. I had signaled delayed dead ball for the obstruction. Now the pitcher appeals the runner missing second base.
OK so I did not allow the appeal or at least I did not call the runner out because of the obstruction. No one really complained but after thinking about this for awhile I'm not so sure I shouldn't have called him out. So what would you folks have done?