<p>I’ve spent many years worrying about this problem so perhaps I can help. We ask rankings to do too many things so it’s not surprising we disagree so often.</p>
<p>But the best way to think of it is to ask “Which variables matter (or should matter)?” and “How should they be weighted?” For the purposes of discussing university quality in research output the three most important are Total amount of quality research, Distribution of quality by department, and average quality of researcher (or per capita if you like). Caltech does really well on the first, pretty well on the second, and spectacularly on the third. You decide on the relative weights. Look at the Shangai-Jiao Tong rankings and see how Caltech’s position bounces around as they flip the weights on per capita citations/Nobels, etc.</p>
<p>Throwing undergrad into the mix you have Quality of Undergrads, Faculty Student Ratio, Quality of Teaching and Training, Spillovers from Research/Reputation Quality to undergrads. Quality of undergrads is itself complicated with regards to number of top quality vs average quality of all students. Looking at the best does better for large state schools. Looking at the average brings Harvey Mudd much nearer the top.</p>
<p>The latter two categories are hardest to judge. Is teaching quality determined by How much the students like their classes or by how well the teachers make sure the students learn even if every one of their students hates their guts? The Spillovers from Research gets to the Big Research vs LAC problem. The smaller a weight you put on spillovers, the better LACs look and vice-versa.</p>
<p>Caltech is in a unique situation because it IS a mega-research palace, but it is also smaller than many/most LACs. That makes any comparisons with standard universities difficult.</p>
<p>Then there is USNWR which survives because of high marketing. It has good variables and ridiculous variables (yield, selectivity measured in percentage accepted, and alumni giving are pointless to me + selectivity needs to be quality of pool adjusted) and totally arbitrary weights which vary from year to year.</p>
<p>The true puzzle is why other big name mags, such as the Economist don’t come up with a high profile but consistent ranking which at least keeps the weights the same from year to year. And then let people decide which variables they favor.</p>