From my point of view, saying they didn't have agency because they were considered property and weren't well educated is not fair.
I'm a white woman too and I do understand fear and limited options. Not having real power on your own. But I also know that some women know how to get their way regardless. And some tried to help or improve the slaves' situation while others used their power because that was what were taught to do. It could be understable under the circunstances, but that doesn't make it right.
Same with some slaves that became violent with the slavers. It is understable and we might not blame them. But at the same time, it can be the solution in the long term.
To me it is fair if black people doesn't trust me by default. Or if they want to consider me complicit. It is true depending on how you look at it.
When I looked to rent an apartment, I had it easier than people of color. I wasn't in a position to not rent to someone racist, so I benefited. At job, many male coworkers have been offered better opportunities just for being men. I don't blame me them for taking them, but it would be nice if they were aware of that edge they had. And didn't reproduce that when they get positions of power.
Also, if we are being used as an excuse to be violent to people of color, we can use our voice to say "not in my name". To be on point, I don't fear black men more than I fear white men. And after seeing stats, it is quite possible that I'm in more danger from men I know than an stranger in the street. So what I do is basically suspect black men in the same situations that I would suspect a white one. That is fair (and to be honest I might mistrust some women too, it is about "dog whistles" XD).