He "took your virginity"? We're not in the 1950's woman! You and he had sex - it was your first time - and by the sounds of it, it was consensual. I hope you used contraception.
Don't go blaming him or playing the victim here. You chose to have sex with him without having defined the relationship. You know that you're both toying with others. If this whole virginity thing was as important to you as you make out, you would have made different choices.
Now, you can't undo what is done. The only option you have is to take responsibility for your decision to have sex. If you don't want to find yourself feeling used again, TALK with him and confirm your relationship status. And if you can't have a relationship without being scared and pulling away, then don't have a relationship till you are ready.
Never regret anything that has happened in your life. It cannot be changed, forgotten or undone. So, take it as a lesson learned and move on.