The rule states, "Charge into or throw an opponent, nor extend the knee to meet the blocker." Or 9-4-2g states "Make any other contact with an opponent which is deemed unnecessary and which incites roughness."
I have a few questions for you?
Was the contact necessary? Was the defender playing football or trying to injure the receiver?
I think you made the right decision. There are many case plays or rulings over the years that suggest he is wrong for his hit. You did the right thing.
Peace