"Sex too soon" usually relates to the idea of destroying a potential relationship. But you don't see yourself dating him or being in a relationship with him - so I'm wondering where the 'too soon' thing comes in. Too soon for what?
I think he's likely avoiding you because your friendship will get all awkward. Sleeping with a friend is one of the best ways to destroy a friendship. It never ends well.
Never regret anything that has happened in your life. It cannot be changed, forgotten or undone. So, take it as a lesson learned and move on.