Goal kick

[Nevadaref]

The Laws of the Game consider a goal kick to be a free kick, so yes, the ball must be stationary when kicked. If it is not, the kick shall be retaken as you have stated.
To be properly placed the ball does not have to be inside the lines, but only has to be partially breaking the vertical plane of the outermost edge of the line. The ball can be on the line or resting just outside of it as long as part of the ball breaks the imaginery vertical plane.
Under the LOTG the ball may be placed and then moved. However, the player/team shall not waste time. This is up to the discretion of the referee. (Under HS rules, once placed the ball may not be moved -- that is deemed time wasting.) Technically, in USSF the caution should be given for Unsporting Behavior (time wasting) or Delaying the Restart because per USSF instruction PI only applies to Laws 12 and 14. (However, there is a sentence in the ATR that gives "Fails to start or restart play properly or promptly, having previously been warned" as an example of PI. Although I don't personally see how that sentence fits with the other as this is clearly Law 13. Chalk it up to one of the many conflicts in the ATR.)
Whether I caution or not at the U13 level would depend upon the skill of the players and the event. Is this a regional tournament, Super-Y league, a state cup, or local league play? I would definitely have a word with the player the first time it happened and inform him that the ball must stop before he kicks it.

PS I will NOT give a player a second caution for this.