It's posible to love somebody and yet do little to show it. Especially after years of being together if relationship starts to stagnate with routine. It still doesn't excuse some of his comments like the one about your father.
But I think this is less about finding out who is right and who is wrong and more about finding a way to make the relationship work in my opinion.
Don't cry, don't regret and don't blame
Weak find the whip, willing find freedom
Towards the sun, carry your name
In warm hands you are given
Ask the wind for the way
Uncertainty's gone, your path will unravel
Accept all as it is and do not blame
God or the Devil
~Born to Live - Mavrik~