People tend to surround themselves with people with similar values, views and interest than them. That means that not sexist men would end in groups of not sexist people and won't interact as much with sexist men.

Take your example, you would down on men who say something sexist, right? So if a sexist men see you do that to other men, how long do you think he would stay around? not long. And he "won't open up" with you about his views on women because he would want to avoid negative repercusions. The way it usually goes is with men trying "dogwistles" and "jokes" to "test the waters". Depending on how do you react, they would show more or less of that sexism.

Think the Pelicot case. Do you think it is coincidence that not a single man in that group wasn't a rapist and denounced? No. There probably was a "screening" by the husband. The web page were he met them probably acted as a first filter.

And again, there are many men who show two faces and have learnt with who they can openly show sexism (or racism or homophobia or whatever) and with who they need to pretend they are not.

Think about the expresion "political correctness". It implies that they act respectful only to pretend they are not as "whatever" as the ones that do it openly. Or think about all the cries about "cancelling culture" or about "wokeness". All those things signal that, in fact, there are sexist or retrograde people trying to make that socially aceptable again.

So yes, there are people who are not sexist or at least try to not be sexist, but there also proudly sexist people. They form groups and surround themselves with people of similar views, try to persuade others to join them. As a result, what you describe of your group can be true while what those feminist complain about can also be true for them. They might be very well surrounded by sexist people because you don't chose your relatives.

What I'm trying to say is that yes, the world is complex but you might want to consider that what they say applies to a significant amount of people. Also, people tend to be "blind" to things that doesn't affect them directly. In the same way that white people might be oblivious to people of color strugles, men many times are ablivious to women's strugles and from women's perspective, looks as if men are ok with that.

Following the Pelicot case, there are less men following it and speaking against those men. Why? Probably because they think it is not them so it is not their business. They aren't guilty and would never do that (true) so what more it is expected of them? (to actively fight the rapist and condem them, but many would feel that is too much).

So maybe the reason feminist, LGBTI activist and antiracism activist have a "bleak and depresing" view of things is because they can't even imagine to simply stand by when those horrible things happens. They know you aren't a rapist and your group isn't either, but expect that you actively join them in condemning the rapist (which high profile cases show not always is condemned).