No.

When someone flirts, there are respectful ways to do it and disrespectflu ways that cross boundaries. Normal people start small and look for reciprocity. Attractive men might get some reprocity, so by the time they touch someone, they know it is consented. Or maybe they risk it and are lucky more times.

"Unnatractive" men might take the "hints", soft rejections or lack of flirting back as a sign of the women not being interested and stop. Those men aren't accused of sexual assault. The ones who are accused of sexual assault, aren't accused of that for being unnatractive, but for crossing boundaries and doing things without consent. The proof of that is that some insanelly attractive actors have been accused of sexual assault. And plenty of regular looking men go around flirting without being accused of sexual assault ever. Because they are respectful.

This is a myth to enforce a double standard so women consent can be ignored.