You can suck it up and try and talk to the guy to let him know where you stand, but it doesn't sound like he will be very receptive to that. Just call your game the way you know it should be called. In the long run, people will notice you hustling and working hard and doing a better job than the guy who's just hurrying things along so he can pick up his check. Don't be afraid to make your calls - otherwise you're not doing anyone any good out there.
Or you can move to a better place to ref...like Portland.