I mean, our relationship was far from perfect (we spent more time fighting than happy),
If more time was spent arguing, than being happy, then you are not meant to be together. You must be pretty young not to realize that. You also must realize that, after 3 years of "love", the fighting didn't get fixed. Love doesn't fix things, action does. And some things will never get fixed. You two are just incompatible, regardless of the emotional attachment you have with him.
Don't you think you owe it to yourself to find someone you can be happy with 90% of the time? Or is your self-esteem that bad? They are out there, but hard to find. That means you have to date lots of people, without throwing away the one who is right for you.
I have a long time interest in psychology, specializing in relationship dynamics for 30 years.
(Please note, we give the best advice we can based on the information given in a post. For better advice, please include the age of all romantic partners.)