I don't buy into the notion that women are more emotionally vulnerable than men. Many men just believe they have to hide their feelings. In some ways, that makes getting hurt emotionally worse for a guy. So seeing that somebody has to say, "I love you" first, I don't see why it can't be the woman.
Being afraid that a guy will hold it against you isn't a reason not to say it either. If anybody is going to hold your feelings against you, then you shouldn't be with that person anyway. Period. So the possibility that the person would continue to use it against you shouldn't exist. Besides, women can be just as verbally abusive as men. In my house, it was always my mother who was verbally abusive and my father was the one who just stayed and took it.
“This planet has - or rather had - a problem, which was this: most of the people living on it were unhappy for pretty much of the time. Many solutions were suggested for this problem, but most of these were largely concerned with the movement of small green pieces of paper, which was odd because on the whole it wasn't the small green pieces of paper that were unhappy.” ― Douglas Adams, The Hitchhiker's Guide to the Galaxy