If this ruins the relationship, it would seem that the relationship is founded on a great lack of communication.
In your other post, you do not understand his sexual conduct. In this post, you are worried that he will misunderstand your contact with your ex. But both of these issues stem from lack of communication. Talk with him.
Having said that, any man would be wise to be wary of an ex who's still hanging around taking money. If you are serious about this new relationship, you would be wise to cut off contact with your ex. Let him find money elsewhere.
Never regret anything that has happened in your life. It cannot be changed, forgotten or undone. So, take it as a lesson learned and move on.