Board Ratings And Rankings

BillyMac · #1 (**permalink**) Sun Mar 19, 2006, 07:48pm

I am presently on the ratings committee for our local high school IAABO board. After using the same ratings and ranking system for over 25 years, we are considering some major changes for next season. At this point we are asking our own board members for suggestions on how to improve our system or suggestions on a brand new system. As a member of this forum, I've decided to ask you, colleagues that represent a wider geographical area, for rating and ranking systems that you may use and believe to be successful.

First, a summary of our present rating system, which is used to assign a ranking that will determine the level of games, and how many games you get for the season:

The present rating system is based on peer rating (80%), attendance at meetings (5%), IAABO refresher exams (open book, group effort, 5%), and availabilty to the commissioner (10%). The peer rating is the portion that we are looking at the closest. Presently, each board member rates every official that he or she has observed in a board assigned game and rates them (one to ten). Varsity officials are required to get to the site early enough to observe at least half of the junior varsity game. Junior varsity officials are required to stay late enough to observe at least half of the varsity game. Most games are two man games, so each of the four officials should give and get three rating for that game. Ratings are kept secret and are sent into the board at the end of the season. During the summer, each official receives a copy of his or her ratings (numbers only, no names of raters, no reasons for the rating), which are used to generate a ranking for the next season's assignments.

How does your local board rate and rank? Do you use a system of peer rating, if so, does it differ from our system? Do you have any specific suggestions (i.e. rating guidelines) to help us improve our system?

I will report the results of this forum thread to the chair and assistent chair of our ratings committee. Thanks for your help.

JRutledge · #2 (**permalink**) Mon Mar 20, 2006, 12:19am

Quote:

Originally posted by BillyMac
How does your local board rate and rank? Do you use a system of peer rating, if so, does it differ from our system? Do you have any specific suggestions (i.e. rating guidelines) to help us improve our system?

I will report the results of this forum thread to the chair and assistent chair of our ratings committee. Thanks for your help.

This is obviously an area thing but we do not have any organizations called a "board." We have official's associations that are optional to belong to. The reality is a little more than half of the officials licensed in the state are not members of any organizations.

The state requires every association to rank the Top 15 Officials in our association for their playoff consideration. We do have a rating system but this is for the state, not for associations to use. This process varies greatly between organizations on how their lists are compiled. For the most part the executive boards come up with their own criteria and those rankings a largely based on past playoff experience. There is one list for Boys and another list for Girls. I only belong to one organization that has very specific criteria for getting on the list and that is a football organization. The rest of the criteria are up to the board to decide the order and who should and should not be on the list. That is why we elect officers to do those kinds of things in the first place.

Peace

ChuckElias · #3 (**permalink**) Mon Mar 20, 2006, 08:59am

Bill, I'm Board 31, just north of you in Springfield. We don't have a ratings system anymore, so I can't really give you any input into that. The college ratings (for ECAC) give the coaches some input; but also include the open book test, peer ratings, attendance at the rules clinic and the assignor's rating.

My question is: why are JV officials rating Varsity officials? I don't think I'd be real comfortable with a 2nd year official rating me.

All_Heart · #4 (**permalink**) Mon Mar 20, 2006, 10:28am

Quote:

Originally posted by ChuckElias
Bill, I'm Board 31, just north of you in Springfield. We don't have a ratings system anymore, so I can't really give you any input into that. The college ratings (for ECAC) give the coaches some input; but also include the open book test, peer ratings, attendance at the rules clinic and the assignor's rating.

My question is: why are JV officials rating Varsity officials? I don't think I'd be real comfortable with a 2nd year official rating me.

Great point Chuck. I wouldn't want to be evaluated by JV officials either. I know that when I was a JV official I was not in the position to tell the Varsity officials how they did. (Even though I thought I was

)

[Edited by All_Heart on Mar 20th, 2006 at 10:31 AM]

Kostja · #5 (**permalink**) Thu Mar 23, 2006, 06:35pm

Hi,

We have a somewhat similiar problem. We are presently working on a new system for coaching and evaluating referees in our local organisation. I have to say coaching our referees and improve their performances is of higher priority than generate a ranking. Yet, we would like to have something like a grade for their overall performance and how well they performed in certain categories, for example communication, fouls, violations (travel, 3 seconds, and so on ...).

In the past we had a standard form for the evaluators, where they could grade the ref in different categories. There was also a place to back up those grades with more details. The problem was, referees don't get better if you just tell them that they had a bad game, but don't give more details. On the other hand, if evaluators give more details (text), it is almost impossible to rank the referees by the different feedbacks we get from different evaluators. What do you do to ensure that your evaluators are on the same page (look for the same things) and that one evaluator/coach doesn't say one thing, and one week later another evaluator/coach says something completely opposite. How do you make sure you can compare reports from different evaluators? How can we train the evaluators/coaches to give proper ratings? This season the worst grade was b/c on an a-b-c-d scale, which means we have very good referees or something's wrong with the evaluations!!! We don't have enough evaluations to calculate an average for every referee, some of them get only one or two evaluations per season.

Any input on this is much appreciated.

Kostja

26 Year Gap · #6 (**permalink**) Thu Mar 23, 2006, 07:57pm

Quote:

Originally Posted by ChuckElias

Bill, I'm Board 31, just north of you in Springfield. We don't have a ratings system anymore, so I can't really give you any input into that. The college ratings (for ECAC) give the coaches some input; but also include the open book test, peer ratings, attendance at the rules clinic and the assignor's rating.

My question is: why are JV officials rating Varsity officials? I don't think I'd be real comfortable with a 2nd year official rating me.

We started doing that last season. As a 2nd year official last year, virtually all varsity officials got superior ratings from me, because from my point of view they were far better than me, though I felt somewhat over my head. But there was a previous problem in our board with JV officials not getting ratings from varsity officials and they figured the goose & gander theory would at least get ratings done. With the advent of the online assignment system and the ability to do ratings online, the assignors could ensure that ratings were given out. This season, after having more expereince and after having learned a lot in camp and through my mistakes, I felt more confident in my observances and the ratings were not all the same for everyone. I stay for the varsity games and not all JV officials do that, so I feel that I am learning even more by observing the varsity officials and watching what they do esp. if they have made suggestions to me after my game. I have thought to myself,"Oh! THAT'S what he/she meant" many times.

So, although the ratings are given by JV officials, I am guessing that for the most part, they are higher ratings than what the varsity officials might give each other because missed assignments etc are much more noticeable to those who have called more games.

zebraman · #7 (**permalink**) Thu Mar 23, 2006, 10:15pm

Quote:

Originally Posted by ChuckElias

My question is: why are JV officials rating Varsity officials? I don't think I'd be real comfortable with a 2nd year official rating me.

How many years of experience would an official need before you would be comfortable? There are some great basketball scouts who never could play a lick. Similarly, you don't have to be a great official to recognize officiating talent. In our group, officials are allowed to take part in the ratings system in their third year. It has never been a problem. What would be a problem is to label those officials as "JV officials" and only let them be rated by the "varsity officials." That would smack of a "good old boy" system and would tend to limit advancement and protect the status quo.

Z

ChuckElias · #8 (**permalink**) Thu Mar 23, 2006, 10:57pm

Quote:

Originally Posted by zebraman

How many years of experience would an official need before you would be comfortable? There are some great basketball scouts who never could play a lick. Similarly, you don't have to be a great official to recognize officiating talent.

No, but you do have to at least understand the rudiments of quality officiating. I'd be willing to bet that 98% of second-year officials don't know enough about positioning, mechanics, or judgment to give an informed rating of an official in a high-level varsity game. So either pick an arbitrary number, like 5 years of experience, or say that Varsity officials are only rated by other Varsity officials.

Quote:

What would be a problem is to label those officials as "JV officials" and only let them be rated by the "varsity officials."

I'm not suggesting that at all. JV officials could be rated by any official who works JV or higher. It just seems to me that a 2nd year official isn't qualified to rate a Varsity official. JMO.

zebraman · #9 (**permalink**) Thu Mar 23, 2006, 11:52pm

Quote:

Originally Posted by ChuckElias

So either pick an arbitrary number, like 5 years of experience, or say that Varsity officials are only rated by other Varsity officials.

I'm not suggesting that at all. JV officials could be rated by any official who works JV or higher. It just seems to me that a 2nd year official isn't qualified to rate a Varsity official. JMO.

I have no problem with requiring an experience factor like years of service before an official can participate in a ratings system (although five years seems quite long since I've seen some great third-year officials) . But when you start saying things like "only varsity officials should rate other varsity officials," you run the risk of creating something of a "good old boy" system where the top guys always rate each other high and keep each other high. That makes it pretty hard for a great young official to break into that group.

Z