Ratings through Arbiter

[Valley Man]

There is a utility that assigners can use to rank officials. You can then rank your games and Arbiter will only list officials of the ranking that you set for a game when you get ready to assign games.

[LRZ]

If I'm reading it correctly, the assigner may choose to have officials evaluate each other:

At https://arbitersports.zendesk.com/hc...to-Z#Officials, it says:

Lead Officials
A lead official is defined by the preference you have set. It can be one of two defaults: The official in the first listed position or the official who is highest ranked in the first listed position. Admins can choose the lead official on a game by game basis by editing the slot and marking it as the lead official. Each game is limited to a single lead official.

All Officials
This allows officials to complete evaluations on each other. Since officials can't evaluate themselves, if an evaluation only has the plate umpire selected to evaluate the plate umpire won't see an evaluation for that game (unless there are multiple evaluations). There is no permission or preference for which officials can complete evaluations. The permission to 'Select games to evaluate' is to allow evaluators to find officials to evaluate.

[BillyMac]

I don't know who set it up, or how it was set up, but we've been using Arbiter to rate our partners for several years. About three times a season, somebody "opens up" the ratings and we can see our ratings, with the names of the raters deleted.

At the end of the year, the partner ratings are combined with trained observer ratings, all being made available to a committee to determine our status as varsity, or subvarsity, with the partner ratings carrying only a little strength in the total rating process.

Also, we tried to use Arbiter to rate everyone at a site, not just partners, but those in the other game at the site as well, subvarsity, or varsity. Arbiter told us that this couldn't be done with their software.

[grunewar]

Our Association uses the Evaluation feature in Arbiter.

We get reminders not to forget to rate officials for games. Criteria on what is evaluated is decided by the Association - I'm not necessarily a fan of our criteria groupings, but, it seems to work ok.

As an official, after score are tallied, you can see your running total and evaluation score averages, and even comments made by other officials. While it is anonymous, if used properly, and directly after each game, you know what the other members of the crew rated you.

Comments are helpful if used as intended.

Personally, while not everyone rates on the same scale, I find it to be a pretty good indicator of areas I need to improve on and a good indicator of how the game went.

You still can't eliminate cronyism and the good ole boy system though.

Bottom Line - we used to have nothing, and this is better!

[BillyMac]

Originally Posted by grunewar

... see your running total and evaluation score averages, and even comments made by other officials. While it is anonymous, if used ... directly after each game, you know what the other members of the crew rated you.

That's why we can only see our ratings about three times each season (including all of your ratings at the end of the season), when the ratings are "opened up", usually for a few days (usually over a weekend) each "open" period, to maintain the anonymity.

Otherwise, you work Friday night, have Saturday off, and a new rating shows up on Monday, making it easy to guess who rated you.

With our system, we just see the ratings in progressive "clumps".

Even then, we have a few guys who "sign" their comments, "Nice job BillyMac. John Doe", or, "Merry Christmas BillyMac. John Q. Public".

Also, we are fined if we don't submit our Arbiter partner ratings in a timely manner.

Arbiter Late Rating – If submitted more than 5 days from game date
- 1 Late or Missed – No Fine
- 2 to 3 Late or Missed - $25 Total
- 4 or more Late or Missed - $50 Total

I find it odd that our board puts such importance, thus the fines, on partner (Arbiter) ratings since they carry such little weight in the overall rating process (how reliable can ratings be when two inexperienced subvarsity guys rate each other in a middle school game, or a freshman game, they barely know if the ball is inflated, or stuffed).

It's the trained observer ratings (these guys show up early to rate both junior varsity officials, and then rate their partner in the varsity game, sometimes they have to make a "nonworking" trip (with some monetary compensation) to rate those who, for whatever reason, may otherwise fall through the cracks) that carry the bulk of the weight when the rating committee determines our status as varsity, or subvarsity, at the end of the season.

We are told that the partner (Arbiter) ratings are only used to supplement the trained observer ratings, that it's mainly the trained observer ratings that determines our status as varsity, or subvarsity, for the upcoming season.

[JRutledge]

A rating system and an evaluation system in my mind are not the same things.

Peace

[BillyMac]

Originally Posted by JRutledge

A rating system and an evaluation system in my mind are not the same things.

Rating often implies a number being associated with one's performance (example, 1-10), which, in some cases, can be limiting. Evaluation, while it may be associated with a number, may involve other forms of measuring one's performance (written commentary, oral commentary, checklists, etc.).

Also, the question pops up, is the "system" used for:
1) Ranking (another number) for assignment purposes (the level and number of games).
2) Education (improvement of officials).