keeper touching ball

[rcwilco]

U13 girls. play is in the penalty area and near penalty mark. there are numerous players from both teams in the box, (about four or five from each team). As the ball is bouncing around from attacker to defender and back, a defender kicks the ball which goes towards the keeper, the keeper bends over and touches it with her hands, when a team mate yells out "Don't touch it!", she takes her hands off of it and straightens up. I have no whistle, the defensive team pauses and the attacking team makes a goal a few seconds later. The goal was scored because of their pausing.
In my opinion there was not intent to kick the ball to the keeper, I do not believe the defender was looking at the keeper, only trying to get it the heck out of there, and the ball was bouncing all over the place. so I had no problems with not calling an IFK.
This was the only goal scored in the game.
After the game the coach and I chatted for a minute and he agreed with me that his team should have played the whistle. He said he was surprised that I did not blow the whistle as other refs have blown it whenever his keeper touches the ball anytime the ball comes from a defender.
I felt bad that in a well played game it was decided by this play but cannot really think of anything to do different. Any thoughts?