Yeah, serious college Philosophy is a shock to many–unpleasantly so in some cases, but some kids actually like it a lot better than they expected. I think it gets some people who swap out of sciences or math because at least some parts of Philosophy actually speak to overlapping interests. But at least in generally rigorous colleges, it isn’t likely to be appealing to the work hard/play hard/pre-professional kids. It is more for life of mind kids who realize analyzing dense texts and crafting arguments is more fun for them than they knew based on their HS-level experiences.
And then some kids learn that is not for them. I would not necessarily say they are not smart enough or that Philosophy is inherently too hard for them. But like most things in a competitive setting, if you don’t like it then it is going to be harder to do well against equally smart kids who actually do like it.