This is so tough: politeness v abuse. I had it with my father-in-law and stood up to it AT THE TIME at the table as it was also a meal time. I said "this is my house and certain things don't happen". He roared off and it never happened again.
So saying, we were also very close and life continued but boundaries and been put in place.
You must speak up and your children will be proud of you.