While I know as officials we have a lot to do on the field/court, and most of the time there is something going on for us, last night, on NBC's Nightly News, Brian Williams reported that in your average baseball game there is only 17:58 of actual "Baseball Action," the rest of the time is just standing around.
For football, he said it was just 11 minutes.
I don't recall him siting the survey source or a definition of "Action", and I'm imagining he's probably talking from a fan's standpoint.