tell him that it should be you apologising to him not the other way round. you need to give him the real reason why you're doing this, he deserves to know. and you need to tell him that you need to talk and say that you can't do this anymore. be truthful but not harsh.