It sounds like you need more in your life and a relationship than just a guy that wants to be around whenever he feels like it. I treated my girlfriends like that in the past and they all got sick of it and dumped me. It was more a selfish act of being with them when I wanted to, and they had to comply. You also want a little something in your life, some romance, doing some public things, sharing the "love" you two have with the world. It doesn't sound like much of a relationship to me.
You care about him but he isn't giving you what you need. If you already told him this and he is being stubborn about it, time to cut the cord. I know it's not easy to meet people being a single mom and I don't know what your dating scene is like, but there has to be somebody out there that likes you and can treat you with respect.
Waking up next to a beautiful girl,
Step outside and say hello to my beautiful world.