If there's no, or not enough, attraction due to an aspect he can't change you're shooting yourself in the foot romantically so that you can either avoid confrontation or preserve your impressions of your own compassion. We're all shallow to some degree, that's what attraction is based on but what we call shallow has a depth all its own. It's not just how they look, it's how they carry what they've got, the walk, the expression on their face... There might be certain criteria for you to be attracted to someone and while you may be somewhat attracted to him, if you have sex with him and don't really want to you're humoring him with pity sex and that's insulting. You have to have that critical mass or its one sided and one sided relationships lead to resentment. Time to break up. You could tell him straight up that he's too short if you give him the respect that he can handle it but I prefer economy with my honesty. If you want to preserve his feelings make up some other excuse to put some space between the two of you and let it drift apart to a comfortable degree. If you think he's a good guy, see if one of your freinds might be interested in him.
Precious and fragile things
Need special handling
My god, what have we done to you?