@ProfSD I agree with the flexibility of being a professor, you pretty much manage your own time, which i think, to some people, more important than the high paying job.
But overall, do you work more than 40 hours per week or less? I know many professors and even though they do have the flexibility that you described, overall they work between 50-70 hours per week. College instructors do have much less hours but professors with a team of graduate students doing research, it’s normal to spend more than 60 hours per week.
Regarding academic job being highly competitive, I think it has always been this way. And there is a very simple explanation - supply vs. demand. I believe most don’t go into academia for money, but for passion or/and prestige. College professor is one of the very few professions where the person doesn’t want to retire, they seem to keep working until they aren’t physically unable to. How many other type of other jobs out there that you often hear “I can’t wait to retire?”, most of them. Plus the tenure thing where you can’t fire someone for under performance like other industries.
My husband’s college advisor and his wife have no plans to retire from Cal and they are close to 60’s. The professor that he TAed for twenty something years ago now still teaching the same class that our daughter is taking (and she still remembers DH!). The guy that wrote the book when we took the class twenty something years ago is still teaching and D is also taking the same class/book from this guy. Our friend is a professor at Columbia who invented a computer language is still teaching there and he is mid 80’s. I can go on and on. If people are not planning to retire, then how many spots left for the younger generations?