You've pretty much stated all the reasons to break up with him, and none for staying with him. It sounds like you know what you need to do, but are afraid of dealing with the pain necessary to do it.
Versus Alason's post here which is pretty much again, the worst kind of advice you can provide someone. The pandering excuse for them to do what the chemical reaction that you believe is your heart says instead of what you really should do.
From a psychological standpoint you're not enforcing your personal boundaries. This is going to lead you to be very bitter, angry, and disillusioned about your life if you don't put your foot down and tell him his behavior is unacceptable.
However it sounds like you lack the personal conviction and self control to do so, in which case simply dumping the sod and working on addressing why you're unable to live your life on your terms is the proper course of action.
"Well, then," the Cat went on, "you see a dog growls when it's angry, and wags its tail when it's pleased. Now I growl when I'm pleased, and wag my tail when I'm angry. Therefore I'm mad."