Rating Systems?
 

A question for those of you who do HS baseball in states where the coaches rate you.

In the state I work in, this year they started a new rating system where coaches rate your work. They also have catagories where you supposedly "need improvement". These ratings are a joke. The catagory ratings are even more of a joke.

How much stock do you put into these ratings and catagories? For example, the number of people who said I need to imporove rules knowledge (or words to that effect, I don't want to be too specific to give the state away) exactly equals the number of games that I called a balk in. I also happen to be a local rules interpreter for my local association, so I know the rules. I guess where I live, if you call a balk, you don't know the rules. The only other time I had a coach say anything to me about a call all year was when some guys didn't think I called the HPB rule right (Come on blue, he's got to get out of the way, he's gotta make an attempt, you know the conversation well enough I don't need to repeat it all for you).

Another rating was for "umpire professionalism". What in the world does that mean in this context?

One one hand, I'm upset at some of the ratings I see. On the other hand, I shake my head in wonder and laugh it off. People who have no clue about mechanics, or even where to stand between innings rate mechanics. Right. It just seems as if you make a call they don't like, they sitck it to you. I've been doing baseball games for a long time, I pride myself on how I carry myself on and off the field and knowledge of the rules. I make mistakes like everyone else, but some of these ratings are nuts.

I would give my left arm to have a guy who knows what he's doing come out and evaluate me. I would be glad to pay someone who knew how to umpire to evaluate me, since that will help me get better. All these ratings and catagories do is give you a bad reputation when it comes time for tournament assignments.

How do you guys feel about these things in your state?

States that have the Coaches rate you are a JOKE.

I find that so ridiculous, I can't even think of a parallel example.

Wait, I know !

Having your Ex-Wife decide your divorce settlement, instead of a judge.

Fortunately, in all the associations I work for in California, all rating is done by senior members of each association.

i can name 5 managers/coaches whose ratings i would buy into if they were given to me. if i was to be evaluated by coaches, which im not, i probably wouldnt pay all that much attention to the ratings. [[edit: my state does rate for HS, as you will see in ajjl22's post. i dont work HS so i didnt really know anything about it so i didnt cover it]] id read all of them and see what was said, but when people get screwed, they like to take others down with them. i would guess that coach evaluations end up in an inverted bell curve -- having a lot of coaches give poor ratings, not that many in the middle, and then a bunch of coaches saying you were good. those coaches giving negative ratings probably got "screwed" on balk calls, so they want to bring you down with them. coaches get "screwed" on the outside corner, so they want to bring you down with them there too. thats not the right way to go about it, obviously, but we see that it is still done.

now as far as ratings from other umpires, there are a whole lot more that i would trust. my association has an evaluation program run by about 20 guys (bob jenkins is one of them) in which you can pay $20 and get 2 evaluations done. they come out to a game that you choose, fill out their eval sheet and leave comments. it doesnt have an effect on assignments, its just between you and the evaluator so that you can get better. something like that is something that i would buy into, and ill be doing so this summer.

Blended system
 

We do both- coaches rate us AND senior umpires rate us. We have crew chiefs, who are supposed to rate you one game and work with you one game. We ask coaches to evaluate us online (the ump evals are paper then entered into a database). It's like pulling teeth to get coaches to do this.

It is subjective. Our coaches get two scratches a year, too, before they turn in their schedules to our assignor. They only get to scratch us at their home places- they can still see us on the road.

Our assignor won't tell us how evaluations affect assignments, so I'm not sure it matters.

Strikes and outs!

Coaches rate officials in the state I live in as well. Not only in baseball, but basketball, football, volleyball, softball also. It's certainly not the best system, but until someone comes up with a better one that's practical, sensible, and affordable, it'll have to do. And recently, our state has initiated an on-line rating system to get a higher percentage of schools to participate. That seems to be working rather well.
I don't put a lot of emphasis on the ratings from the coaches.....they often act on emotion vs. facts and also look at different things to rate that what many officials consider to be helpful.
Local associations are the best source for meaningful rating systems....if we can somehow find enough bodies to make it work.

In Illinois, each coach and is supposed to rate you for each game on a 1 to 5 scale. if you work with a certified official (highest promotion level) they must also rate you, but only once a year.

1 = Qualified to work state final contests
2 = Qualified to work sectional contests
3 = Qualified to work regional contests
4 = Qualified to work varsity contests
5 = Qualified to work underclass contests only

The ratings are averaged and then divided by number of ratings and you are given a percentile ranking. They do this to award officials who work more games.

The problem with the system is that coaches do not do their ratings. I have work 35 varsity games this year. I should have 76 ratings (worked with 6 certified officials) I have a total of 22 ratings. This makes it look like I have worked only 8 games all year.

I think the point needs to be made that the ratings of coaches and officials are only 5 points in a overall 30 point power rating system. Next year there will be another 5 points added to for attending camps. You will get more points for attending a Level 2 camp as compared to a Level 1 camp. I also think in our state we focus way too much on the ratings when the ratings are only a small percentage of what gets us into the playoffs. Also our state is very aware that schools do not rate officials often enough and it might seem to us that this is going to affect us in a negative way. The reality is they are looking for new people to fill spots all the time, especially in baseball because of the lack of numbers. This is why they go out and watch umpires, look for recommendations from associations and if they observe you directly all those ratings will not mean much of anything. I had Anthony Holman tell me directly to have our members on the Top 15 list (in another association) to send the IHSA their schedule for the season. He wanted to take the opportunity to watch several umpires and find people that are not normally getting a shot at the post season. I also know that similar things are being done in other sports as well. Trust me on this, the right recommendation can get you on the radar. Once you are on the radar, it is up to you to prove you deserve to stay there. I know officials in many sports that were rated very high and did not get any playoff games. It was not their ratings that sealed the deal.

Peace

Another thought on the lunacy of Coaches rating Umpires
 

If your going to have Coaches rate Umpires...

I think only a games WINNING coach should submit a rating.

I think we all know that when a team loses, it is primarily the fault of the crappy umpires. Especially that slug behind the plate !:p

In our state this is not at all true. In order to give a rating in our state it has to be done online and the rating has to come with a lot of questions. The IHSA also records if the team loses or wins. They found that the difference between winning and losing was a matter of decimal points. So there was no big difference in a rating from a winning coach and a losing.

Now in our state everyone makes the playoffs (only football you have to qualify for the post season with regular season record). So if they are screwing good officials and umpires, it could come back to haunt them. So I do not think there is much incentive to consider the umpire as the reason you lost a game. As it relates to baseball, we see a lot of the same coaches over and over and they might not agree with a call but they have a history of what you have done over the years or season.

Now I am not suggesting that this applies the same in other systems or other states, I am just stating that they have found this is not true as it relates to our system and we have had a rating system for almost 10 years. I know I liked it better than the previous system when coaches picked umpires they liked or knew and did not have any accountability to who was good and why.

Peace

What's the point of rating systems?
 

There should be two purposes in ratings: (1) constructive, helpful criticism that helps the rated official in particular and officiating in the area in general become better and (2) determining who does what level of game.

I work in a small state, population-wise (Arkansas). I worked for years in basketball and baseball before I gave basketball up- now I'm baseball only.

In basketball before I quit, the ONLY ratings came from coaches. I saw a grand total of three of them in ten years- none of them were particularly flattering or helpful. Our high school assignor told us that these ratings were used in determining regular-season varsity assignments. This did not a whole lot for either purpose, in my judgment, as I didn't find anything in the ratings to build on, and the appointments depended as much on who the coaches knew and liked (politics) as on who would give a quality officiating performance on the court.

In baseball around here, the coaches have no say in who comes to their games other than to get two scratches. That is not to say that there is no politics in baseball around here, just that the politics takes a side seat to who can actually call a game. Coaches still rate. The bottom line is this: unless the coach is an umpire too, (it happens around here) his actual comments about what you did or failed to do are not likely to help you become a better blue. If coaches get a voice in who officiates their contests, to include playoffs, their subjective judgments about "who did me a good job" are more important than your rotation and positioning on R1-R2 with two out from C.

I have the crazy idea that people who know umpiring should choose umpires for big games and playoffs.

Strikes and outs!

Hello everyone, I searched for this topic on the board as I was so steamed and I was glad to find that others feel similar............I had to vent somehow....the following email was sent to the IHSA...I feel a little better after reading your posts.......I am seriously thinking about not renewing my licenses in 4 sports with the IHSA because of this "berating" system. I thought I would share my letter and appreciate everyones insight into this topic.......I am finding blind criticism hardly constructive............


Hello Judy,

Once again....it is I (Official X).....I just reviewed my ratings and once again noticed that someone went in and requested improvement in certain areas....which...as I take it....criticism with no constructive purpose.....the contrusctive part of it would be to point out what it is that was done wrong as alleged by this other official....what I don't like about this is the whole annonymity.....if someone is willing to take the time to go to the computer and imput their opinion of my officiating....then why have they not confronted me at the time of the incident....after the game? I obviously don't agree with this.....I went for promotion this year and made it....but...if you were to look at my ratings....it hardly reflects that I deserve it.....in the previous four seasons....this person...or any other official did not care to rate myself in area of needed improvements...now all of a sudden...in my fifth season, someone has decided to share their opinions of my officiating through the website and not with me personally. I just don't get how this is a useful tool. Does it make any sense to yourself or to Mr. Gannaway? I am sending this to you as I don't have Mr. Gannaways e-mail address handy and your email was at the top of the screen conveniently...if you don't mind...could you share my concerns with him? My whole point is this is hardly usefull information and I find it more discouraging than encouraging...as a fairly new official...I would like to find reasons to continue officiating...if the purpose of this rating system is to weed out officials...then it is doing it's job...as I don't want to continue if this is what I will be subject to. I'd really like to know who it is that is sending their criticism my way and to know what exactly it is that I have done wrong.....I take most offense to the areas of Attitude and Professionalism and the Reaction to Pressure....I can honestly say that I disagree with either of these criticisms...I am my harshest crittic...and I cannot even think of any incidents where this criticism was applicable.......just as I should be held accountable for my actions as an official, I think that those criticizing my performance should held accountable for their criticisms....what was it....when....why was i not approached at the time...after the game......this system....once again......is a farce................I can take criticism when I know who it is coming from........but for someone to hide behind what you have currently set up......is ridiculous....I hope you seriously reconsider or at the very least.....give me a reason why it is necessary to have it the way that it is currently.....

[QUOTE]My HS rating system is also a joke. I have been told that coaches have input except we do not see this input. We get a letter in the mail that ranks each official by a numerical number. That's all we get. We do not know how that number was arrived at etc. It's simply a number.

Personally I did not get HS certified to get the BIG game(s) at the end of the HS year. I got HS certified so that I could umpire in the spring (other leagues than HS) , summer and fall because the contracts call for officials to be HS certified.

You can make a nice "piece of change" umpiring all summer long. You could umpire seven nights straight if your schedule permits. Also, every Saturday / Sunday being at the minimum 2 games a day sometimes 3.

It might be a cop-out answer etc. but I do not put any stock into the rating system. There's politics involved, the "good ole boy networK etc. The point is if you want to umpire in the summer / fall then it's well worth it to get HS certified regardless of the rating system.

Pete Booth

First of all the "need to improvement" section can only be filled in by Certified Official. Coaches cannot even evaluate your umpiring. Secondly, these marks mean absolutely nothing. They only hold value when you give them value. The IHSA does not consider them from all accounts and they do not affect your over all rating. Really, you should not even care. Look at it as a footnote and move on.

Also Judy is not a decision maker in the official's department. She just follows what the policies the IHSA Official's Department has set in place. It is much better to talk to Dave Gannaway personally or to someone on the Official's Advisory Committee that can address this with actual change. You are not the first one to complain about this and nothing has changed. Just be aware for people have talked about this for years at the Officials Conference Delegate's meeting for a few years and the system is likely to stay.

Peace

I don't like it but its better than nothing I guess
 

Our state has been rating the umpires for about three years now and its rediculous.

The coaches are supposed to be "required" to submit it by the state, but in basketball about half of my games were rated and in baseball only about 10%.

If they are going to let the coaches rate us, then we should have to rate the coaches also. Did they control their team, sportsmanship, facilities, security, administration present, etc.,

The sad thing is that most of the losing coaches will turn in the ratings. I laugh because one of the areas is "rules knowledge" and I always get rated less on that area which I would consider my highest.

Last season, we had a "no call" on a fly ball down the first base line. The runner ran to first, and the F1 and F2 came out to get the catch. It was in foul territory and ended up being a drop.

The coach of course said the runner interfered. There was no interference.
Later on my rating coach writes, "umpire definitely doesnt' understand interference"

So basically its all subjective and coaches can say whatever they want no matter how wrong they are.

The state uses the ratings for basketball playoff assignments, but baseball does not.

Thansk
David

In this state, coaches rate umpires on the following criteria:

10% Appearance/Physical
10% Pre-Game
20% Attitude/Temperament
20% Rules Knowledge
20% Mechanics
20% Game/Field Presence

What is Pre-Game, you ask? According to the Arbiter, it includes being "On time for pre-game conference. On time (Enters playing area at appropriate time). Performs all assigned duties: inspections."

And miraculously, I have been marked down in this area. How is that even possible?

We are also expected to rate our partners after each game, a practice I find equally distasteful and which I refuse to do. It seems ridiculous for a rookie to be rating someone who has been doing it for 20 years. (Of course, there are those Smitties who have one year experience twenty times, but ratings would be wasted on them anyway.) Shouldn't these conversations happen in the post-game?

It would be interesting to come up with a system for rating coaches:

20% Attitude
20% Sportsmanship
20% Pulled the pitcher before giving up a grand slam
20% Chose a shortstop who could field a ball cleanly
20% Selected a line-up of players who swing at corner pitches

Have I missed anything?

In an association I belonged to in the past, coaches rated umpires. We found that umpires who had ejections or game ending close plays were rated lower than the other umpires. This occurred to the extent that some D-1 umpires were rated several points below JV umpires working their first varsity games.

Thankfully my current association is more enlightened.

This is the problem the San Diego HS association has. The coaches get to choose which umpires get to work their playoff games. If a close call, whether correct or not, goes against them, they keep that umpire off their playoff list. Or if they are a real jerk, and they get ejected by certain umpires who don't put up with their crap, they only choose the umpires that are weak and let them get away with anything. They really love the umpires who brown-nose them between innings and before the game.

Just go there and do a professional job, don't kiss their butts, and they forget all about you at playoff time. Except the teams that don't make the finals. They always seem to have you on their preferred lists.:rolleyes:

Let me make this clear, our system of ratings is not a local association issue. This is an IHSA system. The IHSA assigns all playoff games not a local assignor. The coach’s ratings are mixed in with Certified Official/Umpires to come up with the same percentage. Coaches can only rate during varsity contests and they can rate you as many times as you work a varsity game for that school. Certified Umpires can only rate you once per year. The coaches/umpire ratings only count for 5 points total (if you in the 90 percentile of all umpires in the entire state) in a 30 point Power Rating system where other things are factored in. And (you heard this here first) they are going to add another 5 points this coming year for the frequency of attending camps along with another category that gives you 5 points for attending a Level 2 (3 Man in baseball and basketball) or 6 hour clinic in basketball for example. So most umpires/officials in our state worry about 5 points which we cannot control and most of the points in the power rating are almost totally in our control (percentage of varsity games, promotion level, open book Part 1 Exam just to name a few). And it has been proven that losing coaches rate umpires/official only percentage points apart from each other which it is not that big of deal. And the Sports Administrators that do the assigning for all playoff contests can make decisions outside of the power rating if they choose to. So things like what you have done while they were watching you might play a much bigger role than what your actual power rating is. You can help yourself or hurt yourself by what you do when observed. I cannot speak for other places, but we worry waaaayyyyy too much about something that is not going to really make that much of a difference. So if you are 90 percent in your coaches/umpire ratings and you are just an "X" as an umpire, you might not get an opportunity that a "C" umpire will get and they are in the 70 percent range.

Peace

And you speak so highly of your former association. Sounds like sour grapes to me. Get on one side of the fence and stay on it.

I wasn't talking just about me, Junior. I've worked plenty of top level playoff games. I started working the CIF playoffs in 1989, and did not have a HS season without playoff assignments through the 2004 season. Don't speak of which you don't know.

I was saying the WAY IT IS in that association. No sour grapes, as I got plenty of great games while there. Unfortunately, this is EXACTLY the way it works here. And sure, I've been requested by many teams too, and often these teams don't make it into the top playoff games (or even make the playoffs). The point is, these games are not given out by merit, but by what a coach thinks is good umpiring (i.e. puts up with his antics).

And for the record, the HS association does not make the rules as to who chooses the upper level playoff assignments. The California Interscholastic Federation makes the rules, and they give coaches a "request list" to fill out, and trust me, many a great umpire has been left off these lists due to not kissing coaches' butts.

I just took you off the list to see what cutting remark you would make to me. It's really a shame that all you can do is insult others. Really speaks highly of your intellect.

Back on the list.

"I would give my left arm to have a guy who knows what he's doing come out and evaluate me. I would be glad to pay someone who knew how to umpire to evaluate me, since that will help me get better. All these ratings and catagories do is give you a bad reputation when it comes time for tournament assignments."

I can help here, I know how to evaluate, and would be honored to have the opportunity to come evaluate you and any others that would would fly me out and pay me to do such. Though since I would be away from home, my fees would be much more than my current association's "mileage only" for eval assignments, say 1/2 pay of the level I'd be evaluating.

Let me know, I would of course need a least a weeks notice to make arrangements with my real job. And our season is wrapping up, with one more big weekend on 6/16 and 17, I am avalable after that.

As a former board member of our association, we reviewed many coaches evals and if they aren't subjective and detailed, you know the ones, "this guy sucked" or "learn the rules" or needs a "stike zone" we would contact the umpire get his/her thoughts on the game and usually wind up deleting the eval, or just gleaning off the useful areas.

As someone else posted, we see a remarkable number of losing coaches submitting evals, with 99% being poor evals, while the victors submit very rarely.

Frusrtation Rules
 

I saw some of my HS ratings posted today. It is an embarassment.

I won't share everything I saw, but this year I did one HS JV game. I got the call an hour before the game to sub for another guy, I show up, do the plate, the losing team gets beat 15-0, and I get a 2 on a scale of 1(bad)-5(good) from the losing coach. the team won like 4 games all year, and dollars to donuts he ripped every umpire in every game he lost. Last time that ever happens.

I applaud my state association for trying to make a better rating system and a more objective way to rate officials for tournament assignments. it is a hard job. The problem is unless you use objective standards like test scores and trained umpires to evaluate people, no rating system is fair to the umpires. Coaches just cannot evenhandedly evaluate umpires, or frankly officials in any sport.

Ridiculous System
 

I find it ironic when any state association or umpire group rely on evaluations by coaches for ratings or assignments. First of all, think of the questions and comments made by HS coaches during the course of a game. " The hand is part of the bat", " But he held the ball for 3 seconds, that means it's a catch ", and I think everybody's favorite," Can I appeal that/ Get help". The point here is most HS coaches have no clue about the mechanics we use, and most are oblivious to the rules.( Example: Break up 2). Most of the successful and respected coaches, if they have a problem with a particular umpire, will call the assignor and let him address the issue. I think evaluations can be an effective tool when used properly. The college group I am in, I think is a good example of how it can be done properly. Our president retired from on the field this season. He will come out, unannounced and observe and take notes. Then after the game he holds a session with the crew, and goes over what was done well, along with areas that need to improve. Veteran umpires, and umpires who have moved up to Pro ball or D I should be utilized to improve things.

Test scores???

Test scores are one of the worst ways to evaluate umpires. I have no problem if they are factored in some way, but test scores only prove someone knows how to pass a test. I have seen guys that are really good at passing tests only to wilt like a flower in a hard rain when the heat gets on. Umpiring is more than passing a test and more than just knowing the ins and outs of a particular rule. We also have to manage people and deal with conflict. Not every umpire knows how to do that very well and it shows when a player or a coach gets on them.

Secondly there is always going to be input from coaches about the job we do. When I worked my first D1 game recently I heard of the report that the coach said to my assignor. When I worked for the first time in a D2 conference I heard about the opinions of what the coaches thought of the job I did. You might think there is not "rating system" in place but the reality there is. At least with a set rating system you might know what goes into it. When you work higher levels you might not ever know what is said behind closed doors and you could be affected more.

Peace

JR, there are not a lot of tools in the tool box here
 

Yes, I agree with you, testing is not a great alternative here. It is your praxsis that matters.

But as a point of fact, how do you determine fitness for umpiring, on the FED or NCAA level?

Here is the list as far as I know, in no order:

1. Evaluation by trained evaluators
2. Coaches' Vote
3. Umpires' Vote
4. Rules/Mechanics Testing
5. Tape Evaluation
6. Reputation
7. Written Recommendations
8. Random Chance
9. Senority
10. # of games worked
11. Other (#of ejections?, player evaluations? I don't know?)

We all agree #1 is best, then possibly #5 is 2nd best. What is #3 on the list?
IMO its testing.
You tell me.

Where is this list coming from?

I do not know what this has to do with our discussion.

Peace

JR,

Sorry, I thought that any list of ways to evaluate umpires was germane to our discussion, as alternatives to a coach's vote. Tersting is not a good way to do it, except maybe as a way to limit the pool of eligible people if resources are not big enough to evaluate a lorger group of people

If I am not contributing to the conversation I am sorry.

Calm down, I just wanted to know if this was a made up list or an actual list used by an organization. This is not about contributing to the conversation (of course you are doing that, this is a discussion board), this is about trying discuss is there a better way to do things. I am just contending that even if you have a list, coaches will still have input on some level. In my state their input is just a small part of the process not the entire process. My state makes it very clear they can consider many factors not in our power rating and they even make a list but suggest they are not bound by those factors. If a particular coach sends a specific report about an umpire, it might be a big factor based on what the administrator wants to do.

Peace

JR,

Spouse in Hospital, that might make it seem I am frazzled. I am calm. It's my own list.

Coaches should have only a little say IMO. In the states I work in, they have way too much say in the process.

I hope everything works out with that. Family members in the hospital are never an easy thing to deal with. Not fun times for sure.

I agree there are places that the coaches make all the decisions. That would be wrong in my opinion. Coaches should not have that much say as to who works and when. But they will have input in just about any system even if it is not clearly stated.

Peace