Yes, in college I knew before Christmas of freshman year who was going to be a top student and who would struggle for the rest of their degree course. It was pretty tough if you found yourself in the latter group, especially because in math the top students were able to work far less hard (if they wanted to), because they could do problem sets in a fraction of the time.
I felt my undergrad degree was the most intellectually challenging period I’ve ever experienced (much tougher than a PhD or anything in my work career). But my kids say they found their college courses easy and were never challenged (except occasionally in sheer volume of work). I think that is more likely due to variation between colleges than between majors.