I've the theory that it has to do more with fear of cis men claiming that they are trans women (lying) to get access to women only spaces.

You are still right that trans women are women and have nothing to do with cis men and that we can distinguis between them. And obviusly trans women should have access to women spaces.

But it might be worth it to shame the cis men who claim to be women, "jokingly" or otherwise in an attempt to disrespect those spaces or abuse laws. Those men are usually transphobe too.

Sometimes identifying the real problem and put it into words is difficult and fear make us behave irrationally.

Hopefully it is a matter of time that we stop thinking like that.