There has to be more to this story and more history. Having a conversation with your assignor and executive board is warranted.
As an assigner, yes there are some blues on occasion (that I will put on a secondary list for 'issues' and not just one or even a few, but over time, its better for the softball association as a whole to not work 'said' umpire if possible) The point is, that before doing something like this, there should be specific instances and documented issues on hand