Sounds like you made the correct call, but your response to the coach was incorrect. You don't have to judge the relative ease with which the fielder can catch a line drive. The infield fly rule only applies to fly balls. A line drive is not a fly ball.
I would have said, "Coach, for me to call the infield fly rule, there has to be a fly ball. That was a line drive."