When they have crossed boundaries. For example, they have touch you against your will. As in grabbing your ass without permision, trying to kiss you against your will (I recomend pushing away from you, not slapping), touching your breast without permision...

If a man does that to other man, he can expect a violent reaction if the man touched doesn't like it. Like a punch. So how is different if it is a women touched?

Now we are expected to patiently explain that touching us agaist our will is wrong. And then, the man who did it can safely ignore it and laught at you. He can even try to argue back about you "really wanted it" even if it is obviusly false. No, his crossing of boundaries was physical and he deserved an strong reaction equal to the infringement. And any sane person would understand that you aren't "merrily slapping men" we are defending ourselves same as they would defend themselves.