First, I wear black compression shorts when I ref (I know, that was more than you wanted to know). That way, if my pants split a little, it's virtually unnoticed. If the split is really bad, I change to a pair of black sweatpants I carry in my bag.
As to what pants to wear, we must wear beltless with slash pockets preferred, no pleats. Most guys get Sansabelt, but I found some in the Penny's catalog that were on sale for 2 pair for under $40. The only problem is that the pocket linings are white, so if you keep a token in your pocket that you move for possession, the lining might pop out when you're moving it and it looks bush.
BTW - I use a roll of mints for my token. My partners are grateful and insist the roll be used up by the end of the game.