If you truly love someone, it doesn't really matter whether or not you are with that person. And it never is difficult to be with someone you truly love in my opinion. But anyway, I'd give my view on "tainted love" which is very common nowadays, and probably in the past too. Heck... love is too great to truly comprehend.
And what do you mean by "being with"? In love? As friends or what? For me it's any kind of relationship.
Anyway, here's my take on it: Because you don't know what to expect. You don't know if you should expect nothing or expect a lot. When you love someone and want to be with that person, you don't know if you should expect that person to feel the same as you or not. If you have no expectations, sometimes you feel that you are not getting any satisfaction out of the relationship cos the giving is always one way. You start feeling bitter - and you start expecting. Humans expect. That's a common human trait. They expect that hard work pays off. People always say that good things come to those who wait and where there's a will there's a way. I say going by how my life has been so far, that's utter bullshit. But that's probably not that relevant here. And sometimes if the person you love loves someone else, you feel very jealous and you love and hate the person at the same time. If you keep thinking too much about this you'd drive yourself crazy. Thus, it gets hard to be with the person you love.
Also, you try so damn hard to impress the person you love. It gets tiring, and gives you a lot of stress. And you want the person so badly, so you're afraid to screw things up. With others, it doesn't really matter that much to you how he/she sees you, cos as far as you're concerned he/she is not that important.
Clarity of mind means clarity of passion too; this is why a great and clear mind loves ardently and sees distinctly what it loves. -- Blaise Pascal (1623 - 1662)