I think that answers your question right there. If there wasn't a violation, you wouldn't call it. I don't see a problem with it later on, especially if you gave him that much leeway. In a close game like that, there may be an obvious advantage gained by this player by camping out in the paint.

It seems the three-second call is one that a lot of officials try not to make. I myself prefer to quietly remind them to keep moving. But if they don't listen to me, they'll certainly hear the whistle.