Frustrated about ratings system and postseason assignment

[SC Official]

Originally Posted by JRutledge

Actually, if you enforce obvious rules, most coaches will respect you more than those that just turn a blind eye. For example, the coach that is always in the coaching box tends to like officials that will enforce the box when coaches are all over the court. Usually, they know these rules are not the ones you as the officials have made up. You just have to be right and you do not have to use the penalty to enforce those kinds of rules.

Bottom line, coaches will always have some say. They may not be the final say, but they have a say and should in many cases. The good coaches know who the good officials are. I would not worry about it any more than I worry about what a coach thinks of the job I do. I do what I feel is right and move on.

Peace

I don't think it's unreasonable that coaches get some say in who they feel the "best" officials are. After all, they are the ones that have a stake in the game. Sometimes I don't think we as officials give them enough credit to be as fair as they can in their evaluations (I'd rather have coach ratings than peer ratings which is what we have in SC). What there needs to be is a "check and balance" to throw out the "extreme" ratings on the high and low end of the spectrum.

But a 100% coach-driven rating system? That might be more ludicrous than the system we have where I live.

[JRutledge]

Originally Posted by SC Official

I don't think it's unreasonable that coaches get some say in who they feel the "best" officials are. After all, they are the ones that have a stake in the game. Sometimes I don't think we as officials give them enough credit to be as fair as they can in their evaluations (I'd rather have coach ratings than peer ratings which is what we have in SC). What there needs to be is a "check and balance" to throw out the "extreme" ratings on the high and low end of the spectrum.

But a 100% coach-driven rating system? That might be more ludicrous than the system we have where I live.

Whatever that supervisor told the officials, it is probably a line that is used to make a point. I doubt the coaches actually pick the officials that deeply. I bet the officials are ranked and that is used by the supervisor as needed. I think sometimes we get caught up in certain things when those things are not as deep or as important as stated.

All systems have flaws. People always complain in our system and coaches ratings are a small part of what we do or how we are considered for postseason games. And one person does all the assigning and he or she can make all kinds of decisions and do based off of things that clearly have nothing to do with ratings.

Peace

[crosscountry55]

This is a very good discussion. FWIW, I hate coach rating systems, but sometimes that's the only way to get a good evaluation sample size if there just aren't enough professional evaluators on the state staff. I would argue that coaches should be expected to submit evaluations for 'x'% of their games. That way, the coaches who don't meet that minimum would have their evaluations removed. And/or....you could exclude coach evaluations whose season averages are > one standard deviation above or below the mean. That way, the guys who always put "5" and the guys who always put "0" are not factored.

I kind of like the approach Rich took. If you stop caring about post-season assignments, you have the freedom to call your regular season games without looking over your shoulder so much. That's very liberating. Worked for me last year as a new guy in a new state when I knew I wasn't going to even sniff the post-season. When I saw a few of my buddies drive 150 miles to work a 1/16 first round blowout with a final score of 76-13 and a paycheck of about $15 more than a regular season game, I realized the Friday night conference rivalry nutcutter I had back in January was a heck of a lot more exciting to officiate.

[SC Official]

It is true that no rating system is perfect. In an ideal world there would be no rating systems and it would be all evaluation-based, but the nature of high school basketball is that there are too many officials, too many schools, and not enough money to send an observer to every game. So in order for there to be at least some attempt at objectivity, that's where rating systems come in.

You could live in South Carolina, where your on-court ability means (literally) nothing for advancement.

25% statewide exam score (closed book, everyone takes in Columbia)
25% peer ratings
25% experience points (5% per year until maxed out after 5 years)
20% administrative (combination of meeting attendance, clinic participation, camp once every three years)
5% cooperation (lose points for turnbacks, etc.)

Most officials, in my experience, don't take the peer ratings seriously (we rate 1-10 in six different categories) and will give every partner a 9 or 10. Most officials receive all the other 50 points. What that leads to is the fact that the statewide exam is basically what determines your position in the rankings. An exam, not your on-court ability, determines where you're ranked as an official.

All varsity games, regular season and postseason, statewide are assigned centrally using the ranking list computed by this formula. There are officials at the top who have no business calling a varsity game and officials in the middle of the pack who are good enough to work state finals. Typically that's because the former group consists of good test-takers and officials whose only redeeming quality is knowing the rules really well.

All this to say, trust me when I say that every system has its flaws.

[Camron Rust]

Originally Posted by SC Official

You could live in South Carolina, where your on-court ability means (literally) nothing for advancement.

25% statewide exam score (closed book, everyone takes in Columbia)
25% peer ratings
25% experience points (5% per year until maxed out after 5 years)
20% administrative (combination of meeting attendance, clinic participation, camp once every three years)
5% cooperation (lose points for turnbacks, etc.)

Most officials, in my experience, don't take the peer ratings seriously (we rate 1-10 in six different categories) and will give every partner a 9 or 10. Most officials receive all the other 50 points. What that leads to is the fact that the statewide exam is basically what determines your position in the rankings. An exam, not your on-court ability, determines where you're ranked as an official.
...

All this to say, trust me when I say that every system has its flaws.

Then that is the problem. The officials have it within their power to do the right thing but are not doing it. Can't blame the system when the system isn't used.

Are those ratings sufficiently anonymous so that an official could give an honest rating without fear of retaliation? It would have to be such that the scores would only become available to the officials after a large number were collected and the system closes.

[BillyMac]

Regular season game (and league or conference postseason games) assignments are based on ratings by a trained observation team supplemented by peer ratings. Each official is rating either varsity or subvarsity and the number of regular season game assignments is based on the discretion of the assignment commissioner, who does a great job.

State tournament games are solely based on coaches votes. The more votes an officials gets, the further that official goes into the tournament. Connecticut has been doing it this way for tournament games for at least forty years, and believe it or not, coaches usually do a pretty good job of selecting the best officials. Every year I look at the tournament list of the thirty-plus officials selected from my local board for the state tournament and only have to scratch my head for one or two of the selected officials, and they're usually gone by the end of the first round.

[Scrapper1]

The only thing worse than not having a rating system for officials is having a rating system for officials.

[JRutledge]

Here is our system and it is simple. We as officials control 4 categories for the most part.
---------------------------------------------------------------------------------

How the Power Rating is Determined: The power rating is a number from 0 to 35, derived from 7 different criteria, as follows:

1. Promotion level: (Officials Control) Certified (C) -- 5 pts., Recognized (R) -- 3 pts., Registered (X) -- 1 pt.
2. Part 1 exam score: (Officials Control) 96 -- 5 pts., 92 -- 4 pts., 88 -- 3 pts., 84 -- 2 pts, 80 -- 1 pt.
3. Previous tournament experience for contests by this gender: state final -- 5 pts., super-sectional or football semifinal -- 4 pts., sectional or football quarterfinal -- 3 pts., football second-round game -- 2 pts., regional or other football playoff game -- 1 pt.
4. Contest ratings (percentile rank of all officials): 90% -- 5 pts., 80% -- 4 pts., 70% -- 3 pts., 60% -- 2 pts., 50% -- 1 pt.
5. Top 15 lists (percentile rank of all officials): 90% -- 5 pts., 80% -- 4 pts., 70% -- 3 pts. 60% -- 2 pts., 50% -- 1 pt.
6. Clinic attendance (most recent clinic): (Officials Control) this year -- 5 pts., last year -- 3 pts., two years ago -- 1 pt.
7. Varsity games worked (percentage of games a school is allowed to play in the regular season in that sport): (Control most of this too) 70% -- 5 pts., 60% -- 4 pts., 50% -- 3 pts., 40% -- 2 pts., 30% -- 1 pt. See table below.

The percentiles for the Top 15 and Ratings used to be things determined by officials as well. Every official's association used to have a Top 15 list, but they took that away a few years ago. Each Certified officials used to also be able to rate officials for varsity contests, but now that is also no longer the case. But just about every category you could control. And there are many other things considered outside of the Power Rating that was considered. I had 35 points last year and did not work the State Finals. There is a policy to not work more than 2 years in a row and then you take a year off. Even if my rating was that high again, I will not likely get back to a State Final. Geography, years of experience, what kind of games you work can all be considered along with many other things. I think people worry too much about things they ultimately cannot control.

Peace

[blue06]

Don't worry about a vote, may be we should vote on if a coach should be in post season! The better officials may not be available or can't get off from their jobs. It is a nice honor if done the correct way. Some times it is just plain ole' politics!

[SC Official]

Originally Posted by Camron Rust

Then that is the problem. The officials have it within their power to do the right thing but are not doing it. Can't blame the system when the system isn't used.

Are those ratings sufficiently anonymous so that an official could give an honest rating without fear of retaliation? It would have to be such that the scores would only become available to the officials after a large number were collected and the system closes.

Everyone in SC wants to do away with the peer ratings, including myself. The problem is no alternative solution can gain enough traction.

They're done through Arbiter for all varsity games. We have six categories where we rate each of our partners on a scale of 1-10 (10 is the best). If you rate below an 8 in any category you have to provide a comment (though theoretically you could just put a dash in the comment box and the system would allow you to submit it). The average score and any comments written about an official are made public to officials shortly after the season ends. You cannot see how you were rated by partner, by game, or who wrote the comment–only the season average score and the comments themselves. The average score is multiplied by 2.5 and applied to your overall rating for the following season.