Practically, the wing officials would seem to have the best point of view, being able to view all of the action at once. I do agree that only the covering official should make this determination.
The best way to prevent any disagreements I've found though, is to simply discuss this situation in your pregame. We do not work with the same crews every week, which has both benefits and drawbacks, but it does force us to discuss these items before every game.
That being said, whether you like it or not, the white hat is on this person's head, so if we're going to be happy little crew-members, we should always go with the opinion of the white hat in the pregame. If he feels he has the call here....give it to him. If he wants you to have it, thank him profusely for the added responsibility and assure him you won't let him down! (or something along those lines
)