My father didn’t like Harvard. But I think OP needs a better idea of what’s wrong. In a few years, he’ll be facing the very “professional” world he’s uncomfortable with now. (Even if he transfers to a state public.) And there is SES diversity at Penn. And wealthy, driven kids at many LACs.
And that Yale prof referred to above is not universally respected as an authority. In fact, many feel he has a grudge. Don’t assume he offers any special insights or truths.
Adding. There are so many clubs at Penn, many that aren’t competitive to get into/dont need to filter you to ensure you have the experience. Many just social, a good range of service opportunities, etc.
If you’re a freshman, this is about a few months ar Penn. Make your own decision. But make sure you’ve explored what really is available.