Ratings through Arbiter
 

Does anyone's organization do its officials' ratings (for coaches/other officials) through Arbiter? Can someone tell me what they need to click to take a look?

(I'm sure there's a setup on Arbiter's end. Just want to know if a demo is viewable.)

Maybe there is a feature somewhere, but I have never seen anything that allows anyone to rate directly through Arbiter. I know some notes can be left for the game by the officials and probably the schools, but no direct rating system that I can tell. That might be something that can be used by the assignor or association that controls their site.

Peace

There is an "evaluations" tab, but, like many arbiter functions, you probably need an assigner to authorize its use.

Search the arbiter site for "evaluations" and you will find several "how-to" articles.

There is a utility that assigners can use to rank officials. You can then rank your games and Arbiter will only list officials of the ranking that you set for a game when you get ready to assign games.

Not Just Assigners....
 

If I'm reading it correctly, the assigner may choose to have officials evaluate each other:

At https://arbitersports.zendesk.com/hc...to-Z#Officials, it says:

Lead Officials
A lead official is defined by the preference you have set. It can be one of two defaults: The official in the first listed position or the official who is highest ranked in the first listed position. Admins can choose the lead official on a game by game basis by editing the slot and marking it as the lead official. Each game is limited to a single lead official.

All Officials
This allows officials to complete evaluations on each other. Since officials can't evaluate themselves, if an evaluation only has the plate umpire selected to evaluate the plate umpire won't see an evaluation for that game (unless there are multiple evaluations). There is no permission or preference for which officials can complete evaluations. The permission to 'Select games to evaluate' is to allow evaluators to find officials to evaluate.

Arbiter Ratings ...
 

I don't know who set it up, or how it was set up, but we've been using Arbiter to rate our partners for several years. About three times a season, somebody "opens up" the ratings and we can see our ratings, with the names of the raters deleted.

At the end of the year, the partner ratings are combined with trained observer ratings, all being made available to a committee to determine our status as varsity, or subvarsity, with the partner ratings carrying only a little strength in the total rating process.

Also, we tried to use Arbiter to rate everyone at a site, not just partners, but those in the other game at the site as well, subvarsity, or varsity. Arbiter told us that this couldn't be done with their software.

Our Association uses the Evaluation feature in Arbiter.

We get reminders not to forget to rate officials for games. Criteria on what is evaluated is decided by the Association - I'm not necessarily a fan of our criteria groupings, but, it seems to work ok.

As an official, after score are tallied, you can see your running total and evaluation score averages, and even comments made by other officials. While it is anonymous, if used properly, and directly after each game, you know what the other members of the crew rated you.

Comments are helpful if used as intended.

Personally, while not everyone rates on the same scale, I find it to be a pretty good indicator of areas I need to improve on and a good indicator of how the game went.

You still can't eliminate cronyism and the good ole boy system though.

Bottom Line - we used to have nothing, and this is better! :p

Anonymity ...
 

That's why we can only see our ratings about three times each season (including all of your ratings at the end of the season), when the ratings are "opened up", usually for a few days (usually over a weekend) each "open" period, to maintain the anonymity.

Otherwise, you work Friday night, have Saturday off, and a new rating shows up on Monday, making it easy to guess who rated you.

With our system, we just see the ratings in progressive "clumps".

Even then, we have a few guys who "sign" their comments, "Nice job BillyMac. John Doe", or, "Merry Christmas BillyMac. John Q. Public".

Also, we are fined if we don't submit our Arbiter partner ratings in a timely manner.

Arbiter Late Rating – If submitted more than 5 days from game date
- 1 Late or Missed – No Fine
- 2 to 3 Late or Missed - $25 Total
- 4 or more Late or Missed - $50 Total

I find it odd that our board puts such importance, thus the fines, on partner (Arbiter) ratings since they carry such little weight in the overall rating process (how reliable can ratings be when two inexperienced subvarsity guys rate each other in a middle school game, or a freshman game, they barely know if the ball is inflated, or stuffed).

It's the trained observer ratings (these guys show up early to rate both junior varsity officials, and then rate their partner in the varsity game, sometimes they have to make a "nonworking" trip (with some monetary compensation) to rate those who, for whatever reason, may otherwise fall through the cracks) that carry the bulk of the weight when the rating committee determines our status as varsity, or subvarsity, at the end of the season.

We are told that the partner (Arbiter) ratings are only used to supplement the trained observer ratings, that it's mainly the trained observer ratings that determines our status as varsity, or subvarsity, for the upcoming season.

A rating system and an evaluation system in my mind are not the same things.

Peace

Evaluation ...
 

Rating often implies a number being associated with one's performance (example, 1-10), which, in some cases, can be limiting. Evaluation, while it may be associated with a number, may involve other forms of measuring one's performance (written commentary, oral commentary, checklists, etc.).

Also, the question pops up, is the "system" used for:
1) Ranking (another number) for assignment purposes (the level and number of games).
2) Education (improvement of officials).

My understanding is that assignors have to rank officials in the Arbiter in order to give them games. I was told these rankings involve what games they can give as an official.

Peace

It's an option, but not mandatory. It is mandatory if you want Arbiter to do auto assigning. You can rate all the officials and rate all your games and then tell Arbiter to fill all the slots.

Sent from my SM-N950U using Tapatalk

You are right, it was for some level of auto-assigning. I do not get to use that feature, but that is the only way I know it was used in Arbiter. And our rating system only involves schools and they do that through the state's website (not Arbiter). That might change at some point, but not right now.

Peace

We have a convoluted, antiquated rating system in South Carolina. 25% of our rating is peer evaluations, which we do through Arbiter. There is an "evaluations" tab and an option to "evaluate officials" underneath it. There are six categories we rate our peers in, on a scale of 1-10. The average is then multiplied by 2.5.

I remember that you said in a different thread that peer evaluation is useless because most people give their peers 9s and 10s. If the system is convoluted and antiquated, what alternative would you prefer that the SCBOA and SCHSL use?

BTW, I do receive evaluations through Arbiter for baseball. There is a standard form that asks the designated crew chief for the game (the person whose name is in bold in Arbiter) to evaluate his partners on mechanics, signals, judgement, procedures, and professionalism, and provide an overall score. The crew chief also is expected to add comments to any rating that he posts. We later get to see how our partners rated us, including an average of our scores in all of the categories, and an average overall score, with their comments.

Not to derail the thread, but since you asked.

The overarching problem in South Carolina is that the state office controls all the varsity assignments for football and basketball. The reason is simply because it has always been this way, and SC is a relatively small state (~200 high schools), so one central contact point isn't too burdensome (i.e. people don't see the reason to change). We have to have a way to rank the officials so the booking office has something to go by. Obviously this system is stupid and I don't need to give reasons. The ideal solution would be for district directors to control all assignments within their districts, not just subvarsity games/holiday tournaments, like is done in surrounding states. Then directors could structure their own evaluation system to recommend their postseason officials. This is extremely unlikely to happen while I am still officiating.

With respect to the peer ratings, yes, very few people take them seriously. This inevitably leads to the statewide exam being the sole differentiator among officials. Currently, the rating system is structured such that on-court ability has nothing to do with one's assignments (which obviously is insane). We have to attend a camp every three years for 5% of our points, and that's it. We need observers to evaluate officials, but there is pushback on this because of cost and paranoid officials worrying about the objectivity of evaluators. I am all for rules knowledge and rewarding officials for getting in the book (and not rewarding officials who refuse to study the rules), but the reality is that we have too many officials working games they shouldn't be working simply because they test the best. It's unfortunate but that is the way it is. Eventually the coaches will raise enough hell that it will have to change, but that has not happened yet.

We do not have "conferences" in SC that have the power to hire/fire their own assigner; we have "regions" within each of the five classifications (1A-5A) that are realigned by the SCHSL every two years, so a college-type system where officials can work for whomever will hire them will not happen here. Most of the private schools in SC compete in a separate league that has their own way of dealing with officiating, so that's an option and many good officials that struggle under the current structure in the SCHSL/SCBOA do very well in the private school league (SCISA).

peer ratings are a waste of time. You either get all 9's or 10's or all crappy scores. It's hardly accurate and depends on whether you partner "likes" you or not. They should be taken with a grain of salt. Coaches reviews are even more worthless.

I do not have an issue with a rating system, just take it for what it is worth. And in my state (Illinois) at least it is only one small factor amongst other things. Ratings are always going to be subjective and often flawed. We used to have a peer system involved with the coach's ratings, but a lot of officials would give high marks. Now I know I would rate what I felt about an official. And only Certified (the highest rank) officials were allowed to do those ratings.

Peace

All I could give a truly accurate rating on would be my partners' communication skills and hustle. I have enough to do on the court to be analyzing their mechanics, signals, positioning, and play-calling.

And I have worked with a bunch of officials who have no business trying to rate me.

Varsity And Junior Varsity ...
 

Agree. Before the most recent change in our rating system, all varsity officials would be required show up early to evaluate both junior varsity officials. Calmly observing the game from the bleachers for two, or three, periods was very conducive to a pretty accurate evaluation. As a retired teacher, who evaluated over three thousand students over the many years that I taught, I had no problems doing these evaluations.

Evaluating my varsity partner is another story, it's always been a problem for me. I've got more important things to do in a game other than observe my partner.

Before the most recent change in our rating system, junior varsity officials were required to stick around for at least two varsity periods and evaluate the varsity officials. Yeah. Sure. A guy with a few years of experience can accurately evaluate a guy with dozens of years of experience working a highly charged varsity game. Tell me another fairy tale. Sure, some of the junior varsity veterans had been to a lot of rodeos, and could possibly do a decent evaluation, but overall, most junior varsity officials, in regard to evaluation, are in way over their heads.

After recent changes, we only do partner evaluations on Arbiter and these don't carry much weight in the overall evaluation process (how accurate can evaluations be when two inexperienced subvarsity guys rate each other in a middle school game, or a freshman game). It's the trained observer evaluations that carry the most weight when comes to one's overall varsity, or subvarsity, status.

I believe that my local board is making a big mistake by not requiring evaluations (on something other than Arbiter, we once had our own locally produced software that could do this) of junior varsity officials by the varsity officials who are already at the site, as we did in the past. These evaluations by veteran varsity officials can be a great resource in a rating system.

On the other hand, one great thing about the recent change is that junior varsity officials no longer evaluate varsity officials. I could never understand such foolishness, and utter nonsense.

Arbiter has an evaluation section. This requires the assigner to assign an evaluator to the game just like he/she assigns officials. Once the evaluator is assign, he/she will evaluate all the officials on the game and when the evaluations are complete the evaluations are:
1. Sent to the officials after completion.
2. Keep an electronic record of the evaluation within the Arbiter system.

Additionally, as an assigner I can review the following information regarding my evaluators:
1. Make sure my evaluators complete all the evaluations assigned
2. Review the average scores, individual scores completed by the evaluator
3. Compare evaluation scores between individual evaluators (make sure they are on the same page)
4. Evaluate the evaluators.

This is a very useful tool for official feedback and evaluator competency.