I could've written your original post almost word-for-word after my first scrimmage. Don't worry, it gets better.
Just wait till your actually working a game, see a foul, and your reaction is "THAT'S A FOUL! Why doesn't someone call it?" before you remember that calling the foul is your job.
All the advice above is good, but I'd especially like to echo the "pick
ONE thing to primarily work on each game" advice. Once you've gotten the hang of one thing (raising your hand when you blow the whistle, for example), then start concentrating on the next thing (Open hand for violation, fist for a foul).
And always remember to SLOW DOWN. As my association's clinician says: once you blow the whistle, the game isn't going to start again until you're ready. There's no reason to hurry to get the ball in play again.
You're lucky in that you've got several scrimmages before you first actual game... use them to your advantage.