I think the concepts of workload, grading difficulty, general stress, core requirements, and academic quality can all get conflated and jumbled into the concept of “rigor”.
I definitely do not believe that top tier colleges are more challenging from a grading perspective. Large publics are notorious for weed out courses for example. And the discussion is very different is you’re talking about Engineering versus English. I would not say English is less rigorous but the required workload is less. Any student can choose a more or less rigorous path through most colleges. The bigger difference is in facilities, resources, possibly teaching quality, and definitely the academic preparation and bent of your fellow students.
I suppose the tippy top colleges also expect you to be more prepared day 1 and may offer fewer pre-Intro level courses.
At the end of the day most students want their degree to be seen as meaning something. Is a degree from UChicago more meaningful than a degree from Brown? Is a degree from Univ New Mexico as meaningful as a degree from UC Berkeley? I don’t know.