I keep my eyes on the players until my partner/partners is/are back in position. Usually by that time a player has brought me the ball, if not, THEN I will go retrieve the ball myself--if it's a rec league game with a running clock then my pace towards the ball and back to the admin spot is, for some reason, very slow
However, I do have a buddy of mine whom I happened to work with last night who refuses to track down any loose balls. He verbally solicits help from the players.
To each is own, but definitely not something that can be penalized.