In my male point of view you said "it's ok" to be casual, so...it's ok!
He seems to enjoy your company, but not enough to be in a relationship with you. He appears to be honest with you and that's very rare.
Most men are just mindless perverts (about 99%) the 1% are the very few who know how to treat a lady when the situation demands.
He's not doing anything wrong, if you start dating many men he will probably back off, it may give the sensation you're a little bitchy. Man and woman are different and a woman that behaves like a man is in no way attractive, besides having VERY LOW standards.
"E ao imenso e possível oceano
Ensinam estas Quinas, que aqui vês,
Que o mar com fim será grego ou romano:
O mar sem fim é português."