My guess is that homophobic/transphobic men aren't worried to be attracted to a trans men. And TERFs aren't worried of having a trans men in the same bathroom/changing room.

It would be funny if it wasn't so sad and harmful. So sorry for that.