Evaluating referees

[Kostja]

I don't know why, but somehow I was chosen by my association to design new forms for the evaluation of referees through veteran referees and/or coaches. Don't get me wrong. We don't let coaches decide, who gets any games, but we like to get their input throughout the season to get an idea what's going on in the gyms.

Now, what's the point of this thread? I'd like to get your input about evaluations in your associations, conferences, where ever you referee, or evaluate other referees. What kind of system do you have to make sure all evaluators are on the same page, and they do not contradict eachother when giving feedback to referees? Do you have any forms where you tell evaluators what to look for (fouls, violations, mechanics, communication, team work ...)? Do you use grades as in school? If anyone could send me one (or more) sample evaluation forms you use or reports you got from an evaluator, that would be much appreciated! Just send me a PM and I will give you my e-mail address.

Thanks a lot for your input.

Kostja

[BillyMac]

RATINGS GUIDELINES FOR EVALUATION

REACTION
Reacts quickly enough to make a decision at the moment of its occurrence.

Makes quick and positive decisions, especially with respect to the “close ones”.

Takes the time to prevent an error from being made.

INTESTINAL FORTITUDE

Remains consistent when calling violations or fouls - without regard to the score, whom it may hurt, or how it may effect future relations with the coach.

CONFIDENCE
Exhibits a confident manner i.e. attention to detail, alertness, firmness, and timeliness of his/her reaction to a situation.

Has a resonant, strong voice that is supported by proper mechanics for purpose of clarification.

POISE
Has a quiet influence on the game that relieves tensions and creates a steady effect upon contestants (both players and coaches alike).

Has control of his/her emotions.

Is courteous and polite.

CONSISTENCY

Is consistent in all calls regardless of situation or point of time in the game. For example consistency in the determination of a block vs. a charge.

JUDGMENT
Uses fair and unbiased judgment and common sense in applying the rules of the game.

COOPERATION

Has the ability to work effectively as a “team” with his/her fellow official

Is not overly sensitive to constructive criticism.

Has a sense of loyalty to fellow officials, a willingness to share the responsibility and avoids attempts to shift the blame.

KNOWLEDGE OF THE PROPER APPLICATION OF THE RULES

Presents a thorough knowledge of the rules of basketball

Appears to make his/her decision with consideration to the effect the calling, or equally as important, the not calling of a rule violation will have. (i.e. advantage / disadvantage)

MECHANICS OF OFFICIATING

Utilizes proper mechanics, up-to-date techniques and procedures as detailed in the Officials Manual.

APPEARANCE AND CONDITIONING

Is in excellent physical condition and exhibits hustle and energy

Official’s uniform and overall appearance is neat, clean and well kept.

[Kostja]

Thanks, Bill! Is this just a Recommendation, or do you demand that evaluators rate/evaluate the referee in every single category you listed?

[AZ_REF]

I'm in the process of setting up a web based evaluation system for my association. It has a form on it as well. If you'd like a login to take a look, let me know.

[coach41]

This is a copy of the online form used by our association to evaulate us. Most evaluators take notes/etc at the game and then go home and submit the form online.



Game Date (m/d/year):

Official's Name: Observer:

Partner (1): Partner (2):

Home Team: Visitor:

Location: Level of Teams:

Quarters Observed (Check one or more): Q1 Q2 Q3 Q4 OT





Please select a rating for the items listed below as follows:
1-poor, 2-below average, 3-average, 4-above average, 5-excellent, NA-not applicable

1. Professional Appearance: NA 1 2 3 4 5 Notes:

2. Court Demeanor: NA 1 2 3 4 5 Notes:

3. Mobility: NA 1 2 3 4 5 Notes:

4. Signals/Mechanics: NA 1 2 3 4 5 Notes:

5. Use of Voice: NA 1 2 3 4 5 Notes:

6. Whistle-Sharp/Loud: NA 1 2 3 4 5 Notes:

7. Partner Communications: NA 1 2 3 4 5 Notes:

8. Pre-Game Activities: NA 1 2 3 4 5 Notes:

9. Positioning-Lead: NA 1 2 3 4 5 Notes:

10. Positioning-Trail: NA 1 2 3 4 5 Notes:

11. Positioning-Center: NA 1 2 3 4 5 Notes:

12. Press Coverage: NA 1 2 3 4 5 Notes:

13. Game Situation Awareness: NA 1 2 3 4 5 Notes:

14. Control of Players: NA 1 2 3 4 5 Notes:

15. Control of Coaches: NA 1 2 3 4 5 Notes:

16. Judgment: NA 1 2 3 4 5 Notes:

17. Advantage/Disadvantage: NA 1 2 3 4 5 Notes:

18. Consistency: NA 1 2 3 4 5 Notes:

19. Off Ball Play: NA 1 2 3 4 5 Notes:

20. Post Play: NA 1 2 3 4 5 Notes:

21. Rules Application: NA 1 2 3 4 5 Notes:

22. Points of Emphasis: NA 1 2 3 4 5 Notes:

General Comments:

[Kostja]

Originally Posted by AZ_REF

I'm in the process of setting up a web based evaluation system for my association. It has a form on it as well. If you'd like a login to take a look, let me know.

@AZ: That would be great. Can you send me the login via PM or do you need my email address to set up an account for me?

@all: It takes a while to translate it, but I will post a copy of the form we have used so far, soon.

[AZ_REF]

Due to popular demand I've created a login for everyone on the evaluation website I've put together. This is in early stages of development and is far from perfect. And I am NOT a web designer, so no *****ing about the layout. That will all be improved with time.

First off head to http://www.tboaonline.org and click the Members click here link. On the login page the username is forum_tester and the password is whistle.

After you login you'll see the menu at the top of the screen. Fancy logo eh? You can ignore the change your info and change your password button. Also the list of officials is people in my association and their contact information. Where you'll be intrested is the New Evaluation button. 2nd down on the right. Click that and your presented with a list of officials you can evaluate. Pick any of them.

Now you're filling out an evaluation. No field is required but they are all recomended. You can check more then one level if your evaluating them on more then one game. For instance if you watch a Freshman / JV doubleheader and head in at half to get ready for the varsity game. You'd select freshman girls, jv girls. In this instance you more than likely observed 6 quarters. The varsity quarters observed is how many quarters the Freshman / JV official (the person you're evaluating) watched you.

The rest of the form is pretty self explanatory. Fill it out and press submit. Now your presented with a spreadsheet that you can print off. When the system is complete it will also email this form to the person you evaluated and whoever is in charge of evaluations in the association. The vote at the bottom will not be sent to whoever you evaluated.

Click the back to the menu link. There's a few more buttons worth looking at. The evaluations you've turned in and evaluations of you are important. Click on them to get a list of who you evaluated and a list of evaluations of you. You can go back and look at any of these evaluations.

The evaluation summary is only visible to board members. It contains a breakdown of evaluations and yes and no votes to move up. If you click on a name you get a breakdown of who voted which way and on what game they voted that way.

If anyone has any questions or problems let me know. The system is still really new and a work in progress.

[jkjenning]

Very nice beginning, AZ_REF!! - thanks for allowing a tour!
My $0.02:
1. Have a logout
2. Provide a wider selection of ratings -> "1 through 5", for instance, rather than only three.
3. Default to 'No Selection' or something other than 'Exceeds Expectations'
(If there are no 'T's during the game, for instance, how can you provide a rating?)
4. Are there categories that the questions can fall into? (providing categories of questions might be easier to navigate than one long list)

Otherwise, very easy to use and having a Comment box for each question is good!

I'm not in any position to be evaluating other officials, but I know from past evaluations in other situations that I would be hesitant to rank "Exceeds Expectations" for about anything - we were always urged to "get 5's" from our customer evaluations and I was always thinking how I would never rate someone with a 5 unless they really were flawless - and they can't be flawless every time! I think I would prefer a system of ranking that ranged from one through ten, and expect that the best officials would tend to average ratings of about eight. Besides, if an officials "Exceeds" my expectations three times in a row, then by the fourth time they have only "Met" my expectations - but they are still no less qualified to work that level.

[AZ_REF]

Originally Posted by jkjenning

Very nice beginning, AZ_REF!! - thanks for allowing a tour!
My $0.02:
1. Have a logout
2. Provide a wider selection of ratings -> "1 through 5", for instance, rather than only three.
3. Default to 'No Selection' or something other than 'Exceeds Expectations'
(If there are no 'T's during the game, for instance, how can you provide a rating?)
4. Are there categories that the questions can fall into? (providing categories of questions might be easier to navigate than one long list)

Otherwise, very easy to use and having a Comment box for each question is good!

I'm not in any position to be evaluating other officials, but I know from past evaluations in other situations that I would be hesitant to rank "Exceeds Expectations" for about anything - we were always urged to "get 5's" from our customer evaluations and I was always thinking how I would never rate someone with a 5 unless they really were flawless - and they can't be flawless every time! I think I would prefer a system of ranking that ranged from one through ten, and expect that the best officials would tend to average ratings of about eight. Besides, if an officials "Exceeds" my expectations three times in a row, then by the fourth time they have only "Met" my expectations - but they are still no less qualified to work that level.

Thanks for your input. I can't believe I forgot a logout, oops.

About the rating system. Our old form was a 1-5 job. However the board found that it wasn't working too well. We developed this form on the basis that for each level an official is at we expect certain things. For instance a 4 (first year official) our expectations for Communication w/ partners is lower than what we would expect of a 2 (experienced frosh/jv official). We are in the process of putting together a list of what we expect to see in each level. I personally like the 1-5 thing myself, oh well.

When it's all done it will default to N/A, but for something I was testing it was easier to leave that option out. I like the idea of categories, just not sure the best way to implement it. Our old form was in categories and if you think about it the areas fall into a logical order on the online form. I'm not sure if I'd rather add a title before each section and keep it all on one page or move to a multipage system and have a few categories on each page. Thanks again for all you input it's appreciated.

[tjones1]

Very interesting tour AZ, thanks for allowing us to see the site.


However, I would disagree that jeans are "ok" for a JV game.

[SMEngmann]

AZ, I belong to Coach41's association which has a similar evaluation system to the one I demoed on your site. The problem I have with both systems is that both try to be so inclusive in their categories that you get bogged down in minutia. There are some 30 or so categories to score, so an evaluator cannot do a good job evaluating each of them, and others simply require little/no thought. When an assignor is looking at this evaluation, he/she could easily lose the forest for the trees by simply seeing a wave of numbers, some of which are more important than others.

My second point, and there will probably be disagreement, is that I don't think judgement should be listed as a category for evaluation. Judgement is a subjective category and there's no way of quantifying that simply with a numberical or descriptive grade. Many aspects of "judgement" cannot possibly be taken into account by an observer because the observer doesn't have the whole picture. We can measure definitively whether an official was in the proper position to make a decision, whether that official is believable and effective with his/her mechanics and what type of demeanor the official projects when on the court, but there is no way, without getting inside the official's head, you can measure judgement. Further, it is not fair, without the benefit of reviewing the tape with the official, ideally, to determine whether that official simply missed calls, or has good or bad judgement. There's no way you can evaluate judgement without knowing what specifically that official saw or without knowing the context of a situation.

I do want to differentiate between judgement and other aspects that are measureable, or can be evaluated. For instance, game management, including bench, table and players can be observed without questioning judgement. How did the decisions this official make pan out? What could he have done differently? Also, foul selection is another measurable. Is the official calling obvious fouls? Is the official calling things that aren't there or looking for "stuff" to call? All of that can fall under a game management category, and comments on it would be, in my opinion, very valuable. But rating and questioning judgement is, in my opinion, the wrong way to go. Focus primarily of positioning and mechanics, and keep it short and simple (unlike this post )