I don't know why. I find it more crude and insulting than erotic. And in conversation with work buddies I have to force myself to use it because using the word "vagina" seems too...uptight? And other pet names are too cutesy for a construction environment. It's not that I find it offensive (unless someone is calling me a pussy, I suppose), I just find it, unappealing, I'm having difficulty explaining.
I don't have as much of a problem is dick, or cock, though I almost never use them when referring to, or engaging in sex with my g/f.