Partly, people tend to associate prestige with selectivity factors. In the examples of the schools you named, they vary when compared across colleges nationally from about 37th for UCB – through 42nd for UM, 56th for UVa and 74th for UCLA – to 88th for UNC, at least when considered by narrow measures at the time of this analysis:
https://amp.businessinsider.com/the-610-smartest-colleges-in-america-2015-9
Since this seems to comport with what you’ve heard, the perception of UNC may arise from an objective basis.