Loose your indicator. It is a crutch, and as you can see, it is not a very reliable one.
Head in the game - remember the activities that led to the outs.
When someone asks that question ("Hey Blue, how many outs?") it always feels like they want an immediate answer - the average umpire looks at his clicker-indicator-thing-a-ma-bob and rotely reads it to the requestor. And then starts questioning himself as to whether it was the correct answer or not.
I don't use an indicator. I've lately taken to spending the time to get the correct answer. I recall the plays and where the outs occurred. (Additionally, I will try to remind myself by indicating the number of outs before every - other new batter.) Confirm with my partner. Confirm with a scorekeeper that is actually old enough to know he has 10 fingers and is paying attention to the game.
Crap happens.