Did I mess things up?

[1confusedgirl]

Ok...this is my first time posting here and I am in desperate need of some analysis from others....I am wondering if I screwed things up in some way or what?

So a little background information...we are both single parents. He has an ex wife who up and left him (her choice) and kids some time ago, however she is back in the picture now, but does not have custody of kids....where as I am a completly single parent. We have been talking for several months, have seen each other on a monthly basis. Our kids even met once. But this weekend was different....we were having an "us" weekend. But.....there were multiple obstacles that started the weekend, guilt from his ex that she had to cancel her plans and have the kids, he had to rent her a car since she has no vehicle....and then to top it off, he coaches his son's baseball team and due the lack of coaches (there was only 1 at the game the other day) his son ended up getting hurt and the ex called and blammed him for it! Which in turn made him feel terrible I know that she knows how to get to him, and I do realize that she is the children's mother and will always be, but it seems that she is just making it difficult for him on all levels-unless she needs something from him-something to her benefit (like a ride to or from work due to her lack of not having a vehicle).
So now if I haven't lost anyone yet....back to my dilema....we were having a great weekend together, went out with friends one night, went home had a great night in bed, woke up with a hangover-stayed in bed all day together (which was so great) until he got the phone call about his son and found out his kids were staying the night at a babysitters house which he was not happy about either....this is where he became detached yet seemed to try and fight or at least hide it but during a moment....he told me that he didn't feel I was into the situation?? I tried to explain in words how I felt about him...but it was not enough for him.....he ended up deciding to go home-which left me feeling sick to my stomach as if I wasn't good enough, didn't make the right choices and I am really afraid this is going to ruin any future our relationship might have and I don't want that because I really care about him and his children. He is a great guy all around and I don't want to loose him from my life. I can accept that I did not initiate any of the actions, which maybe I should have and I always end up regretting not doing this when we are apart but I really just don't know what to do and don't want him to base our realtionship on this one instance.
We did talk last night as he was driving home, and then this morning he text me to tell me he was getting his kids and so I returned his text by asking him to call me later on when he had time because I told him I would really like to talk more. Is that too upfront? Should I let the situation go? Is it over? Can someone please give me some insight...anything....any feedback please?

Any advise on what I can say or do at this point?