When I find myself saying to myself, "Ya' know, that howler monkey was right."
Seriously, I agree with the post above that said it's time to quit when it's no longer fun. That's the main reason (along with exercise and the fulfillment of my passion that there is great value to our society in youth sports programs) that I officiate.
I almost hung it up two years ago, but I went to watch a few games and got "that feeling" back.
Of course, when I have to get up and down the court in a walker, that might be a tip-off.