wrong server

[sport]

Incidence: I as the down official believe that Team A has a wrong server going to the line, I am fairly sure that the wrong server is approaching the serving area, I ask the excellent and very experienced score keeper if the correct server is at the line, she replies yes, server scores two points, I ask again, "are you sure that is the correct server?" score keeper replies "yes". server makes another point, on the next serve the ball is lost. Team B serves with the correct server, loses the ball, Team A sends same server to the line as was up previously, I say to the score keeper, "so she was the wrong server last time?" score keeper replies, "oh my yes". Coach on Team B becomes incenced, because he heard me asking about serving order previously. Game goes on, no points were taken away because the serve had changed. What could or should I have done? Should I have taken time to check the score book and make sure?