Does he include you on these outings with other women, or does he try to keep you separate? And if you DO go, how is his behavior? Is he affectionate with you in front of people? (meaning, does he put his arm around you, sit close to you, etc.)
If he acts like he's a single, available guy when he's around them, I might be worried. If he is just hanging out and it is clear he is into YOU, I wouldn't stress over it.
Relax... I'll need some information first. Just the basic facts - can you show me where it hurts?