Honestly, in addition to what indi is saying (absolutely correct), I WOULD be worried about your brother's sexual boundaries. Most child victims know their offenders, and very frequently, it is their mother's boyfriend. For this reason, I don't advocate single mothers dating until their kids are pretty much grown unless they keep them pretty much separate.
Yeah, I'm paranoid, but no one's going to hurt my babies because *I* made a mistake in judgement.