Frustrated about ratings system and postseason assignment

[crosscountry55]

This is a very good discussion. FWIW, I hate coach rating systems, but sometimes that's the only way to get a good evaluation sample size if there just aren't enough professional evaluators on the state staff. I would argue that coaches should be expected to submit evaluations for 'x'% of their games. That way, the coaches who don't meet that minimum would have their evaluations removed. And/or....you could exclude coach evaluations whose season averages are > one standard deviation above or below the mean. That way, the guys who always put "5" and the guys who always put "0" are not factored.

I kind of like the approach Rich took. If you stop caring about post-season assignments, you have the freedom to call your regular season games without looking over your shoulder so much. That's very liberating. Worked for me last year as a new guy in a new state when I knew I wasn't going to even sniff the post-season. When I saw a few of my buddies drive 150 miles to work a 1/16 first round blowout with a final score of 76-13 and a paycheck of about $15 more than a regular season game, I realized the Friday night conference rivalry nutcutter I had back in January was a heck of a lot more exciting to officiate.

[SC Official]

It is true that no rating system is perfect. In an ideal world there would be no rating systems and it would be all evaluation-based, but the nature of high school basketball is that there are too many officials, too many schools, and not enough money to send an observer to every game. So in order for there to be at least some attempt at objectivity, that's where rating systems come in.

You could live in South Carolina, where your on-court ability means (literally) nothing for advancement.

25% statewide exam score (closed book, everyone takes in Columbia)
25% peer ratings
25% experience points (5% per year until maxed out after 5 years)
20% administrative (combination of meeting attendance, clinic participation, camp once every three years)
5% cooperation (lose points for turnbacks, etc.)

Most officials, in my experience, don't take the peer ratings seriously (we rate 1-10 in six different categories) and will give every partner a 9 or 10. Most officials receive all the other 50 points. What that leads to is the fact that the statewide exam is basically what determines your position in the rankings. An exam, not your on-court ability, determines where you're ranked as an official.

All varsity games, regular season and postseason, statewide are assigned centrally using the ranking list computed by this formula. There are officials at the top who have no business calling a varsity game and officials in the middle of the pack who are good enough to work state finals. Typically that's because the former group consists of good test-takers and officials whose only redeeming quality is knowing the rules really well.

All this to say, trust me when I say that every system has its flaws.

[Camron Rust]

Originally Posted by SC Official

You could live in South Carolina, where your on-court ability means (literally) nothing for advancement.

25% statewide exam score (closed book, everyone takes in Columbia)
25% peer ratings
25% experience points (5% per year until maxed out after 5 years)
20% administrative (combination of meeting attendance, clinic participation, camp once every three years)
5% cooperation (lose points for turnbacks, etc.)

Most officials, in my experience, don't take the peer ratings seriously (we rate 1-10 in six different categories) and will give every partner a 9 or 10. Most officials receive all the other 50 points. What that leads to is the fact that the statewide exam is basically what determines your position in the rankings. An exam, not your on-court ability, determines where you're ranked as an official.
...

All this to say, trust me when I say that every system has its flaws.

Then that is the problem. The officials have it within their power to do the right thing but are not doing it. Can't blame the system when the system isn't used.

Are those ratings sufficiently anonymous so that an official could give an honest rating without fear of retaliation? It would have to be such that the scores would only become available to the officials after a large number were collected and the system closes.

[BillyMac]

Regular season game (and league or conference postseason games) assignments are based on ratings by a trained observation team supplemented by peer ratings. Each official is rating either varsity or subvarsity and the number of regular season game assignments is based on the discretion of the assignment commissioner, who does a great job.

State tournament games are solely based on coaches votes. The more votes an officials gets, the further that official goes into the tournament. Connecticut has been doing it this way for tournament games for at least forty years, and believe it or not, coaches usually do a pretty good job of selecting the best officials. Every year I look at the tournament list of the thirty-plus officials selected from my local board for the state tournament and only have to scratch my head for one or two of the selected officials, and they're usually gone by the end of the first round.

[Scrapper1]

The only thing worse than not having a rating system for officials is having a rating system for officials.

[JRutledge]

Here is our system and it is simple. We as officials control 4 categories for the most part.
---------------------------------------------------------------------------------

How the Power Rating is Determined: The power rating is a number from 0 to 35, derived from 7 different criteria, as follows:

1. Promotion level: (Officials Control) Certified (C) -- 5 pts., Recognized (R) -- 3 pts., Registered (X) -- 1 pt.
2. Part 1 exam score: (Officials Control) 96 -- 5 pts., 92 -- 4 pts., 88 -- 3 pts., 84 -- 2 pts, 80 -- 1 pt.
3. Previous tournament experience for contests by this gender: state final -- 5 pts., super-sectional or football semifinal -- 4 pts., sectional or football quarterfinal -- 3 pts., football second-round game -- 2 pts., regional or other football playoff game -- 1 pt.
4. Contest ratings (percentile rank of all officials): 90% -- 5 pts., 80% -- 4 pts., 70% -- 3 pts., 60% -- 2 pts., 50% -- 1 pt.
5. Top 15 lists (percentile rank of all officials): 90% -- 5 pts., 80% -- 4 pts., 70% -- 3 pts. 60% -- 2 pts., 50% -- 1 pt.
6. Clinic attendance (most recent clinic): (Officials Control) this year -- 5 pts., last year -- 3 pts., two years ago -- 1 pt.
7. Varsity games worked (percentage of games a school is allowed to play in the regular season in that sport): (Control most of this too) 70% -- 5 pts., 60% -- 4 pts., 50% -- 3 pts., 40% -- 2 pts., 30% -- 1 pt. See table below.

The percentiles for the Top 15 and Ratings used to be things determined by officials as well. Every official's association used to have a Top 15 list, but they took that away a few years ago. Each Certified officials used to also be able to rate officials for varsity contests, but now that is also no longer the case. But just about every category you could control. And there are many other things considered outside of the Power Rating that was considered. I had 35 points last year and did not work the State Finals. There is a policy to not work more than 2 years in a row and then you take a year off. Even if my rating was that high again, I will not likely get back to a State Final. Geography, years of experience, what kind of games you work can all be considered along with many other things. I think people worry too much about things they ultimately cannot control.

Peace

[CJP]

I think a good way to chose post season officials is to do it while all stakeholders are sitting at the same table. AD's and coaches can all sit in the same room and pick from the pool of available officials. Coaches or AD's can nominate who they feel are the best and any objections should be discussed. The minutes of that meeting should be made available to the officials. Objections should be noted in detail so the official can use the feedback to get better.

I think doing it in a open forum will eliminate some of the politics.

Evaluations from coaches are not a bad thing but I would not allow them in the decision process to chose post season assignments. I would welcome feedback anyway I can get it.

[JRutledge]

Originally Posted by CJP

I think a good way to chose post season officials is to do it while all stakeholders are sitting at the same table. AD's and coaches can all sit in the same room and pick from the pool of available officials. Coaches or AD's can nominate who they feel are the best and any objections should be discussed. The minutes of that meeting should be made available to the officials. Objections should be noted in detail so the official can use the feedback to get better.

I think doing it in a open forum will eliminate some of the politics.

Evaluations from coaches are not a bad thing but I would not allow them in the decision process to chose post season assignments. I would welcome feedback anyway I can get it.

I do not mind getting ratings from coaches, but not an evaluation. They know nothing or not enough about officiating as much as I or most of us would know little to nothing about actual coaching. We know coaches do not like officials for some silly reason or for following the rules, so why would I want to know what a coach thinks of me that I have little or no respect for?

And your suggestion would still be political. It would make people realize who does not like them either way and likely be seen as held against one or the other. Then if we had a coach that was on the record not valuing our ability, then anything we did with them moving forward would be held against us even if we did not really care or pay attention. There is a reason politicians have closed-door meetings about policy. The public knowing how the sausage is made only would cause a problem in many cases for the person making the comments or the result to be accepted.

Ratings are fine but do not tell me that a coach knows when I should rotate or who actually should have made a call or not. They often think we are all looking at the same play when we clearly are not.

Peace

[blue06]

Don't worry about a vote, may be we should vote on if a coach should be in post season! The better officials may not be available or can't get off from their jobs. It is a nice honor if done the correct way. Some times it is just plain ole' politics!

[SC Official]

Originally Posted by Camron Rust

Then that is the problem. The officials have it within their power to do the right thing but are not doing it. Can't blame the system when the system isn't used.

Are those ratings sufficiently anonymous so that an official could give an honest rating without fear of retaliation? It would have to be such that the scores would only become available to the officials after a large number were collected and the system closes.

Everyone in SC wants to do away with the peer ratings, including myself. The problem is no alternative solution can gain enough traction.

They're done through Arbiter for all varsity games. We have six categories where we rate each of our partners on a scale of 1-10 (10 is the best). If you rate below an 8 in any category you have to provide a comment (though theoretically you could just put a dash in the comment box and the system would allow you to submit it). The average score and any comments written about an official are made public to officials shortly after the season ends. You cannot see how you were rated by partner, by game, or who wrote the comment–only the season average score and the comments themselves. The average score is multiplied by 2.5 and applied to your overall rating for the following season.