If you're going to wait for him, then wait for him with the TRUST that he just needs to take care of some personal stuff before he can commit to you fully. If you don't think you should wait for him because you don't TRUST him or what he is saying to you then just sever all ties now so that you can get on with your life while getting yourself over him.
Do you trust that these "issues" are ligit and that he is capable of overcoming them?
Your brother had a lot of negative things to say about you being in a relationship with this guy. Just what were the negatives? I think they may be relative to the advice you're going to get... as would knowing what these so called "issues" are he needs to overcome before he can be with you.
- - - Updated - - -
I'll add to the above by also asking you if you could get anymore cryptic yourself when he wanted to know when you got off work. You don't know what he wants because you don't know how to communicate what you want. Why didn't you just make a time for when he could "talk" to you if you didn't know when you were getting off work? How non-committal you are and you wonder why you don't understand where his head is at.
- - - Updated - - -
Further: Don't sleep with him again until AFTER he fixes his shit. You'll give him ZERO reason to hurry that up and get it done if you do.
“The willingness to accept responsibility for one’s own life is the source from which self-respect springs.” ~Joan Didion