Evaluation Criteria - Gimme 5

[Larks]

I received this e-mail from an AD. I worked one game for this particilar school and work about 20% of my games in this league.

Dear VIT: (ok, I added the VIT part)

I am looking to develop an evaluation system to be used to evaluate officials at games at SCHOOL Z, and possibly to get implemented into the LEAGUE X which currently has no evaluation system. As it stands now, there is no concrete way of evaluating any officials, and I feel some good young officials are being overlooked in our league. An evaluation system being used by all LEAGUE X coaches could provide valuable data to help improve officiating and to help young officials get the chance to move on to the varsity level.

What I am looking for is 5 criteria to evaluate officials on. I would like to get input from coaches, AD's and officials and develope a rating system that is simple enough to be filled out quickly by a coach and sent to one person for compilation.

Over a season, there would be enough data collected from several sources, and this would give an accurrate picture of the officials abilities. One bad game would not neccessarily hurt an official because there will be several other games to evaluate also.

Thank you for your help in this matter

SCHOOL X AD


So...old Dawgs:

Do I even touch this with a 10 foot pole? This feels political to me.

If I do respond: What are 5 evaluation points in your opinion that make the most sense. Remember, this is to be completed by a coach.

Thanks

Larks

[kmref]

I was invovled in a similar situation:

Write/call AD and ask him/her for the ten things that they might look for in a "good official". Weed through the answers and you will probably be able to come up with five that are reasonable. If not, you can "lead" them in the right direction.
After you have [reviewed] the five criteria send it back with your approval. You have just reversed the responsibilty from you to them and they are the ones that can take credit for the evaluation process.
It worked very well for me. Just an idea to keep you out of "political" mix.

[zebraman]

Dear A.D.

It has been proven that those who have a vested interest in winning and losing make for very subjective evaluators in regards to referees. Although a perfect ratings system for officials has never been invented (and probably never will be), the most successful ones combine independent observers with peer ratings.

Thanks anyway.

Ref

[pizanno]

Gee...is that AD an official as well? Sure is interested in young officials.

Rather than implement a full-on rating or evaluation system for coaches, who may not be the most objective or qualified, you might suggest a ranking sheet for each school.

Have each coach to rank their top officials (we use 10), as well as to list any officials they absolutely do not want to see (ding list).

There are no criteria or guidelines, and the process is entirely subjective, but it gives coaches an opporutnity to give feedback and helps assignors become aware of preferences.

Many of our coaches don't care and don't even respond.

[LarryS]

I’m not sure I would pass on this, but I’m also not sure I would take an active role. How’s that for decisiveness?

If it were me, I would respond as follows…it’s long

Dear School X AD:

I applaud your desire to develop a fair and equitable method of evaluating the officials in your league. I also believe you are correct in your assessment that many fine young officials are being overlooked. That said, I am not sure a system with only 5 criteria to be used by coaches, AD’s and officials will serve your purpose. While it is probably a safe assumption that coaches and their AD are looking/wanting the same things from the officials calling their games, other officials may look at other factors in determining the skill level of an official.

For example, we would probably all agree that an official should have excellent rules knowledge. That said, everyone must understand that the rules of basketball may change from year to year. It is sometimes difficult for officials to fully understand and apply the changes at the beginning of the year. That would probably be amplified when a coach or AD (who may have never coached basketball) is attempting to rate a person in this area. That is not intended to be a criticism of non-officials, it is simply a matter of your duties as an educator and coach taking the time we use for training. I’m sure you will agree that there are many misconceptions related to the rules, which supports my concern.

Another factor that needs to be considered is the mechanics used by the official. That is not limited to the mechanics of reporting fouls, but also applies to pregame duties, positioning during the game, staying within their primary area of floor responsibility, etc. Those may be easy matters to evaluate if someone is concentrating on the official at all times. However, the coach has the primary responsibility of coaching the team and directing their actions. How many times do you not even notice the official until the whistle has blown? He or she may have taken several steps between the time the whistle has blown and you find him on the floor. Accordingly, you would have no way of knowing if he was in the best position.

Obviously, judgement is key to the success of an official. This could also be called “rules application”. That would be difficult for anyone to evaluate because nobody will have the same angle or line of sight as the official making the call. In theory, half of the calls made in the game will be 45 to 50 feet away from each coach. Properly evaluating the official’s judgement will be difficult at best from that distance, especially with other people moving between the observer and the calling official.

Things that could be evaluated fairly easily are:
The level of hustle exhibited.
The intensity that person puts into a contest.
How they handle/control the activity on the floor (i.e. rough play).
Do they interrupt the flow of action?
How do they handle questions/comments from coaches and players?
How do they handle tense situations involving the coach?

An additional concern I have is, unlike officials, a coach and AD have a vested interest in who wins the contest. Human nature being what it is how would you factor in the emotions of the evaluator? Given that everyone involved is an individual, how do you factor out the interpersonal portion? An official may not like a coach personally, but he can still fairly call his team’s game. Conversely, a coach may not like an official. That does not mean he is not an excellent official.

If a fair and equitable evaluation process is your goal, I think serious consideration should be given to the schools in the league contributing an additional game fee to hire a former or current official to serve as evaluators. By taking this approach, most of the personal factors would be removed. The problem of the coach having to divide his or her attention to properly observe the official would be eliminated and you would be guaranteed to have someone with a complete understanding of the rules and mechanics of calling a game. If you feel this is farther than you can go at this time, I will be happy to review the factors your league coaches determine to be the most important, help you define the parameters of each and possibly assign them a weight.

[CYO Butch]

While I understand why officials don't like being evaluated by coaches, I think your concerns are probably unwaranted. Sure coaches can be biased, but for every one who dings an official because they lost, don't you think there will be one who praises them because their team won? Nobody likes to be second guessed, particularly by people who do not have the knowledge and training of the people they are second guessing, but that's something we all have to live with all the time. A bad official can influence how a game is played, but rarely can change an outcome. A single questionable call can change an outcome, but I have never seen a game with a single questionable call. If an official is having a bad game, it is likely to show up more than once. If the official is simply not very good, it WILL show up more than once. But I have never, never seen an official who had a bias that intentionally favored one team.

As a coach, I might be inclined to downrate officials who were pretty lenient in what they called under the basket because I coach girls and because I don't value the rough stuff inside highly. Conversely, my evaluation of an official who called a lot of contact would probably be pretty high. If my perspective were the dominant one in the league, then officials who called it closely would get ahead, while the others would get down rated. Altogether, I think that is fair.

Frankly, my bet is that the major thing influencing how a coach perceives the officials is their game management. How bad is that?

[pizanno]

CYO Butch: your eval would be welcome. Your presence on this site is testament to your commitment to quality officiating.

How many of your clipboard collegues frequent this site?

[LarryS]

CYOButch,
You made one of my points for me! "I might be inclined to downrate officials who are pretty lenient in what they called under the basket because I coach girls and because I don't value the rough stuff inside highly. Conversely, my evaluation of an official who called a lot of contact would probably be pretty high." So, if the official is not calling the game the way YOU want him to call it, he gets a bad evaluation? Remember, not all contact is a foul. That is the point I was trying to make with the personal preferences of a coach.

I would have no problem being evaluated by a coach. In fact, since my chapter allows a coach to scratch an official I am being evaluated by coachs every game. Am I convinced all the coaches can be fair and competent...No. Am I convinced they are all unfair and incompetent...No. But how can a coach effectively do his job and evaluate the officials at the same time? I know I cannot fairly assess a coaches skills while calling a game. I have too much to do.

Also, I mentioned a situation where both coaches yelled for a foul to be called on the same play. By definition, one of them clearly did not see the play or misapplied their knowledge of the rule. They cannot both be right when the want opposite results. It is also probable that neither had the proper angle to evaluate the play since it was on the baseline and they were near midcourt.

A prime example of built in bias on coaches. My daughter has competed for 2 national championships this year in cheerleading (different governing bodies). In the first competition, their coach was telling the parents the team won after the routine...they finished third. In the second competition...the coach was telling the girls they probably didn't place well so prepare yourself for the announcement...the national championship ring will arrive in a few weeks. In both cases, the coach saw something totally different than the judges when watching the action.

Who was wrong? Maybe neither...they had different perspectives of the routine. The coach thought the first set of judges were jerks because he felt they were dead on...if you feel like your team is playing great defense but a lot of fouls are going against you, will you be able to fairly assess the officials judgement. My daughter's coaches could not. You may be better.

You obviously care about the sport and the level of skill shown by EVERYONE on the court. But I think you will have to admit that not all coaches can be fair. There are bad coaches and bad officials. Let me ask you a simple question, and this in not intended to be critical. How many coaches could pass a test on the rules of basketball? I know the rules, but could not effectively coach a team. There are many excellent coaches who do not know the rules.


[Edited by LarryS on Feb 7th, 2003 at 03:04 PM]

[pizanno]

Originally posted by LarryS
Also, I mentioned a situation where both coaches yelled for a foul to be called on the same play...They cannot both be right when the want opposite results.

Did somebody say "BLARGE"?!

...are you an attorney, Larry?

[LarryS]

Originally posted by pizanno

Did somebody say "BLARGE"?!

...are you an attorney, Larry?

Nope...I'm a bean counter (accountant)

[CYO Butch]

Originally posted by LarryS
CYOButch,
You made one of my points for me! "I might be inclined to downrate officials who are pretty lenient in what they called under the basket because I coach girls and because I don't value the rough stuff inside highly. Conversely, my evaluation of an official who called a lot of contact would probably be pretty high." So, if the official is not calling the game the way YOU want him to call it, he gets a bad evaluation? Remember, not all contact is a foul. That is the point I was trying to make with the personal preferences of a coach.

Actually, I'm just admitting that everyone has some kind of bias. Not all biases are wrong or bad. I am biased toward honesty, sportsmanship, etc. Many of the things we call "values" are our own cultural biases. That doesn't make them bad at all. But it does make it difficult for those who don't share those values to understand us. I know if I were asked to evaluate officials, I would go out of my way to try to be fair and even handed. I would do my best to avoid letting "personal" things intrude. But I also know that I believe that girls basketball is better off when the officials' calls about contact are tighter, acknowledging that not all contact is a foul. I have a lot of reasons for this belief, among them the fact that (in my experience) girls take contact from another girl more personally than boys do. Also, we (male) coaches tend to be pretty protective of our girls, so tempers can flare if we think our girls are getting hurt physically because of "no calls". As a result, to me, a good official is one who does his or her best to keep the contact under control. I do not doubt that however hard I might try, this view of girls basketball that I have would probably influence how I evaluated an official, even if the traits I'm talking about were not amongst those being formally evaluated. If this "bias" of mine is offset by another coach who likes to "let them play", so be it. The norms of the league will determine the results.

Your question about how many coaches could pass a rules test is a good one. My guess is that the number is pretty low. In my league it's probably higher than the norm because the league issues a rule book to each coach at the beginning of each season. I'm pretty sure that I could pass it, partiuclarly after hanging out here, but I'm kind of anal that way. I think all of us should know the rules well enough to pass the test because that would help us to be better coaches. It might also help shut up some of the howlers. (Like you said about coaching though, I know that I am a terrible ref when I do it during our internal scrimmages, and I would be a joke if I actually took the court. That job is just too tough!) I'm also smart enough to realize that whether I'm right about the rule in a given sitch, and the official of my game is wrong, the official is always right.

Of course a coach can evaluate the officials during the game. Isn't that what they're doing when they're howling? Seriously, your point about that is well taken. The only time I notice that an official's mechanics are bad is when something is getting so messed up that it draws attention to it. I would never advocate that coaches be the only ones to evaluate officials. That would be just plain dumb.

I don't thik the cheerleading example carries to basketball very well. As I understand it, cheerleading is "judged" according to a number of factors, but many of them are highly subjective. While basketball officials "judge" virtually everything they see during the game, the do so at a very fine level of detail that has been expressly defined to reduce the amount of purely subjective judgment involved. I would place cheerleading judging along side figure skating judging as subect to highly variable personal interpretation. Heaven help us if basketball ever becomes THAT subjective.

[LarryS]

I'll admit the cheerleading analogy was kinda lame...but it was all I could comeup with at the time (especially since I just wrote the check for the ring). But you got the point I was trying to make...a lot of coaches (just like fans and parents) sometimes see with their heart more than their head. I would guess that when you girls have a great week of practice and then hit the floor a lay an egg you are less critical of the officials (figuring your team is just playing badly...when in fact both your team and the officials are having a bad night.)

The most important question is why can't I get coaches in my games that are as sharp as the coaches on this board?

[CYO Butch]

Originally posted by LarryS


The most important question is why can't I get coaches in my games that are as sharp as the coaches on this board?

Back at ya. Why can't I get officials who care as much as the ones on this board? I guess it does sort of self-select, doesn't it? (I might pass on a couple of your guys, however. There seems to be some pretty strong anti-coach bias in a couple of you. No offense MP.

Seriously, I have seen a wide range of officials over MANY years, and to date there has only been one that I have any problem with. His is an affable guy, but he's in his 70s, can't or won't run, and is never in position. I have never had a problem with his calls because I've never seen him make one. Even then, I'm sure he does a better job than I would.

[Nevadaref]

If that AD really wants to see good, young officials, the first thing that they need to do is throw out the seniority system.
This more than anything else is what puts people on games that they shouldn't be working and keeps people from doing those games who should be there.

[Jurassic Referee]

Originally posted by Nevadaref
If that AD really wants to see good, young officials, the first thing that they need to do is throw out the seniority system.
This more than anything else is what puts people on games that they shouldn't be working and keeps people from doing those games who should be there.

I really don't think that there is anything the matter with a type of seniority system,as long as:
1)the officials involved have achieved a certain level of officiating expertise.
2)the officials are still physically able to keep up with the play and get into position,and are actually doing so.

Experience is a very valuable commodity to have as an an official.Problems do occur,I think,when an official is lacking something in #'s 1 & 2 above,but he/she is still being thrown out on the floor ahead of better qualified officials,whether they are younger or not.I also think that you have to throw your good,young officials into the fire,to get them the experience that they need so that they will be ready to replace the older guys/gals when they they start wearing down. JMO.