Or not, plenty of not "buff" men have success with women. Example: Elon Musk. He is not fit, he is intelligent and some women like that. Each men have their own qualities, body is just one.

And many women find DiCaprio when he was young hotter that Swazenneguer, who is way buffer. Hint: different women like different things.