Maybe it is enough to acknowledge that there was centuries of inequality and that we are far from it. So when you consider the "now", you can include the past that is responsible for the present we have. People inherit both things and ideas from their parents. Men have inherit a system that still benefits them. So remembering the history (those centuries of opresion) is what helps to udentify the reasons of certain inequalities in the now.
I don't blame germans for the Nazi crimes, but neither I forget the story, so when I see similarities on the current far right, I can point them out. Same with sexism. Men now benefit from customs that comes from that inequal past.
By the way, you analogy is interesting, because if a Nazi killed jews and stole their riches and his son inherit those riches, should we acknowledge that the son, a German that never killed jews benefit from the nazism? I mean, he is rich in the present but wouldn't be without the riches stolen by his nazi father. Do I blame him? Depends on what he does with that inherited money. Does this answer your question?
In all this discusion you seem to believe that "the past is the past and has no influence on the present". But that is not true. The present can only understood if you understand the past. And the present has inherited many things from the past. Right now there is still sexism and racism. So when women bring the past, maybe try to understand why and how is that past influencing the present and might be creating inequalities now, not "billions of years ago". Because in the same way that we have neonazis and far right in the present, we have never lived in a world without sexism.
So you can start by answering yourself. Why do you think that we have brought up the topic in this specific conversation? Because I did explained the reason I was bringing that up.
Or to use your example about indigenous people. Lets say that you are explaining a racist situation or a power imbalance where the white people have the upper hand over the indigenous people and bring up that since white people wiped indigenous at that point, now the white people have inherited the properties stolen generations ago and are rich, while indigenous people doesn't have as much properties and that creates inequality now. And instead of trying to understand that and at least acknowledge that, I complain and say "but I didn't do that! why do you bring up something that happened centuries ago?". Do you think that the defensive answer does something for equlity? No. It is just a way to NOT speak about that and for me to continue the status quo.
Now, what can I, a white but not rich person that hasn't inherited riches from that genocide do? Well, I can admit that truth and by doing that create pressure for the ones who in fact have benefit and still benefit (Since they inherited the riches) from that. So they try to offer a compensation. Or maybe put in place help in the form of public scholarchips or services that compensate for the inequalities.
Sometimes you are not required to do a lot. Sometimes just listen and try to see the other person point of view is enough. Women aren't holding you responsible for the past, but might mention the past to explain the "whys" for the inequalities now.
So do we allow the germans to forget about the nazis just because now most of them are now? Or are we able to hold acountable the nazis, remember the past and still have some german friends?