Most people will tell you what you want to hear, but since I don't know you, I am telling you what I *really* think, which is probably similar to what your former brother-in-law is thinking.
You said your parents could have lost everything they worked for, which implies they had assets. Any accountant would have told them how they could access their money.
Another option is that you could have worked an opposite shift of your father, since I am quite sure your mother didn't need the both of you caring for her. Your son could have still been around when you were with your mom.
Meh, doesn't matter... Water under the bridge. Anyway, this is probably the way her family is thinking, too. I'm sure they also have more "dirt" on you.
Maybe they will simply be happy for your ex-wifes newfound happiness, and forget all about you.