First of all, your thread title threw me a little..
As to your question, I would agree it might be better if you used your peripheral vision as the main way to watch the defender's position relative to the ball handler and whether or not the ball handler picks up (or starts) a dribble, which would affect your count. IOW, your total area is the main focus of your vision with the two players not being your primary focus, but a secondary one. After all, you don't want to miss a foul, like an illegal screen, while you're counting five seconds.