The subject has been written about:
https://www.psychologytoday.com/us/blog/wander-woman/201107/why-women-have-care-about-their-looks
https://www.psychologytoday.com/us/blog/the-human-beast/201305/why-women-feel-bad-about-their-appearance
https://www.psychologytoday.com/us/blog/the-human-beast/201612/why-women-spend-so-much-effort-their-appearance
But there is some of this concern among some men (note: gym members and gay men were overrepresented in the referenced study):
https://www.theguardian.com/lifeandstyle/2012/jan/06/body-image-concerns-men-more-than-women