Wilson Chrome Moliben mask

[gotblue?]

I think the problem is caused by a slight interference between the edge of the bill of the hat and the upper horizontal bar/wire at the top of the eye slot. The mask is "flatter" (low profile) than most other masks, which brings that bar closer to the hat. In removing the mask, if you don't clear the bar from the bill, the bar will start lifting the hat, and the harness will usually finish the job.

Slightly loosening the side straps of the harness helped me to pull forward enough before lifting in order to clear the bar away from the bill of the hat, which led to success for me. You said that you already have a fairly loose harness, so loosening further may not help. Possibly working on pulling forward more at the start would help. The two-handed approach has also worked for me, for some reason that is what I do at the end of a half-inning most of the time.