I think the best - albeit perhaps not the most helpful - advice is to use your judgement.
From reading your post, I think you're doing the right thing. I ref a few 5th and 6th grade level games and I have a son who plays at that level. When I've had a coach complain, I've explained myself (quickly and in the course of the game), and that worked. If it doesn't, you may need to use the "T". I haven't used a T at this level, but I've seen it done and it was warranted.
For example, I just had a 6th grade girls game where Coach A is saying "3 seconds, 3 seconds..." My partner and I have already talked about it and are telling the young ladies (who don't have the ball) "get out, get out" whenever possible. Then A1 holds the ball for about 5 seconds in the lane and my partner calls 3 seconds. Coach A tells me we have to call that both ways. I replied, "Coach, we're looking both ways. Being lost in the lane and holding the ball while you're in there are two different things." By rule was the coach right? Yes, there were a ton of 3 second violations in the game. But at that level specifically, I'm using my judgement on calling it. (About 1 minute later I got a 3 seconds on B1 who was holding the ball in the lane. Last 3 seconds call of the game.) We didn't hear from Coach A any more.
On the other hand, PLEASE don't ignore violations / rules that clearly give one team an advantage. An example of this is when Team A (5th grade boys) fights there way back into the game after being down by about 10 in the second half. They tie the game with 3 seconds left. Team B inbounds...dribbles...horn...ball still in hand...shot...it goes and the ref (whom I know) counts it. Team A is devastated, most of players crying in disbelief. Everyone knew it shouldn't have counted. Ref says to me later, "Yeah, I know it wasn't good. But the junior high girls were waiting to practice."