OK, it's time for you to enforce some boundaries. It's time for the "I love you and your kids, but their behaviour is unacceptable to me and I will not continue with you if things don't change".
You MUST put yourself and your own child above the relationship with him. Think forward to when his boys are 17 and 16....if they are out of control now, imagine what they'll be like when they are teenagers. Not to mention that your child will grow up seeing you being disrespected and think it's OK. Make no mistake, these older boys are role models.
Never regret anything that has happened in your life. It cannot be changed, forgotten or undone. So, take it as a lesson learned and move on.