I think it's a legitimate complaint. Have you already moved in with him and that's why you need the closet space? If yes, since he said he will remove them, I would just remove her clothes into a box myself and put mine in. If you haven't moved in yet then I would hint at doing the above when you are there. His angle is practicality not emotional attachment, this would work from the same angle of practicality.
Don't cry, don't regret and don't blame
Weak find the whip, willing find freedom
Towards the sun, carry your name
In warm hands you are given
Ask the wind for the way
Uncertainty's gone, your path will unravel
Accept all as it is and do not blame
God or the Devil
~Born to Live - Mavrik~