How do we evaluate the rankings?

onlinepass
Sat, 24 Mar 2012 04:10:28 +0000

i.e. What should we make of the rankings?

There are pros and cons to all rankings.


ARWU Rankings [|ARWU 2010](http://www.arwu.org/ARWU2010.jsp)
Advantage: data is objective
Disadvantage:
(1) the biggest flaw is that the rankings measure research output quantity, not research output per faculty member, which means that large institutions with similar research quality with smaller institutions will be ranked much higher. The authors admit to this point, as well as other academic and professional commentators. Want proof?
(2) The weights might be adequate because citations, prize winning alumni or faculty, publications in main
US News [Best Graduate Schools | Top Graduate Programs | US News Education](http://grad-schools.usnews.rankingsandreviews.com/best-graduate-schools)
Advantage: it takes into account a school’s undergraduate admissions difficulty
Disadvantage: randomness of criteria. “peer-assessment” is not an objective ranking strategy. Endowment per student or professor also is proportional to the size of an institution. the randomness of the weights given to these criteria are questionable as well.
QS and THES
[QS World University Rankings - Topuniversities](http://www.topuniversities.com/university-rankings/world-university-rankings/2011)
Advantage: it takes into account a variety of factors, from citations to so called “educational quality”. says it controls for the size factor.
Disadvantage: those variety of factors are somehow misunderstood by the rankers. public sector funding for research heavily favors large schools, and also does not distinguish graduate from undergraduate institutions. Also favors UK schools for some reason. Also, “employer score” or “international faculty ratio” are no measures of how good a school is at all. Employers favor a variety of student types. Some companies favor state school graduates, while other companies favor ivy league types.


RePEC Economics Departments Rankings
[Rankings at IDEAS: US Economics Departments](Economics rankings: US Economics Departments | IDEAS/RePEc)

Other good rankings include RePEC economics rankings – they only deal with objective data such as citations per faculty and then popularity scores (which has to do with academic trends).

Revealed Preference Rankings of undergraduate departments
[A Revealed Preference Ranking of U.S. Colleges and Universities by Christopher Avery, Mark Glickman, Caroline Hoxby, Andrew Metrick :: SSRN](http://papers.ssrn.com/sol3/papers.cfm?abstract_id=601105)

I wanted to include the revealed preference rankings about the cross-admit data, but then the rankings only measure school choice and prestige viewed by a high school senior. Nonetheless, since high school seniors would like to be in the school that sets them best in graduate schools or career or at least they want to be surrounded by the best students, so the rankings say something about the “prestige factor” of an undergraduate portion of a school.
Shortcoming is that it’s not an international ranking of schools so it’s less applicable to other rankings.


The New York Times rankings 2005 in terms of graduate school admissions, a good indicator of undergraduate student quality.


Conclusion:
I think some combination of a size-controlled ARWU with the revealed preference rankings will be the ideal university ranking system.

*
ARWU rankings by per-capita performance score (from the 2010 data) can be used as a school’s research quality indicator, as well as a graduate school ranking (most of the time)

[|ARWU 2010](http://www.arwu.org/ARWU2010.jsp)
1(1) California Institute of Technology* 100
2(2) Harvard University* 69.2
3(3) Princeton University* 65.5
4(4) Massachusetts Institute of Technology (MIT)* 64.5
5 Ecole Normale Superieure - Paris 56.7
6(5) University of California, Berkeley# 54.2
7 University of Cambridge 53.1
8(6) Stanford University* 50.1
9 Swiss Federal Institute of Technology Zurich 47.1
10 University of Oxford 41.1
11(7) University of Chicago# 40
12(8) Cornell University* 38.1
13 Karolinska Institute 38.1
14(9)University of California, San Diego 37.9
15(10)University of California, Santa Barbara 37.3
16(11)Yale University* 37
17 Rockefeller University 35.6
18 The University of Tokyo 34.5
18(12)Case Western Reserve University 34.5
20(13)Carnegie Mellon University# 34.3
21(14)University of California, San Francisco# 33.6
22(15)University of Colorado at Boulder 33.5
23 The Imperial College of Science, Technology and Medicine 33
24 University of Goettingen 32.9
25 Leiden University 32.4
26 University of Copenhagen 32.3
27(16)Columbia University* 32.1
28(17)Brown University* 32.1
29 University College London 31.6
30(18)University of California, Los Angeles 31.2
31 Moscow State University 31.2
32 The Australian National University 31.1
32 Kyoto University 31
34 University of Munich 30.7
35 Technical University Munich 29.2
36 University of Heidelberg 28.6
36(19)University of Pennsylvania* 28.5
38(20)Northwestern University * 28.4
39 University of Helsinki 28.2
40 University of Utah 28.2
41(21)University of Washington 28.1
42 University of Toronto 27.9
43(22)University of Illinois at Urbana-Champaign# 27.8
44(23)The Johns Hopkins University# 27.2
45 University of Zurich 27
46 University of Melbourne 27
46(24)University of California, Irvine 26.9
46(25)Washington University in St. Louis# 26.7
49 Uppsala University 26.6
50 The Hebrew University of Jerusalem 26.5

Some rankings of the University of California campus schools are a little bit questionable, but I’d say that they are good medicine research schools.


schools ranked at the top 20 in the revealed preference rankings.

schools ranked above at least the top 40 in the revealed preference rankings but not in the top 20 revealed preference rankings.



So to choose the best schools overall, in terms of both research and undergraduate student body, in the order of revealed preference:
[A Revealed Preference Ranking of U.S. Colleges and Universities by Christopher Avery, Mark Glickman, Caroline Hoxby, Andrew Metrick :: SSRN](http://papers.ssrn.com/sol3/papers.cfm?abstract_id=601105)
2 Harvard University* 69.2
1 California Institute of Technology* 100
16 Yale University* 37
4 Massachusetts Institute of Technology (MIT)* 64.5
8 Stanford University* 50.1
3 Princeton University* 65.5
28 Brown University* 32.1
27 Columbia University* 32.1
36 University of Pennsylvania* 28.5
38 Northwestern University * 28.4
12 Cornell University* 38.1
44 The Johns Hopkins University# 27.2
11 University of Chicago# 40
20 Carnegie Mellon University# 34.3
43 University of Illinois at Urbana-Champaign# 27.8

Other schools that are good but are left out: NYU, Duke, LSE
Depending on said goal of the individual student one might also consider schools like:
Dartmouth and/or top LACs which are as good as, or better than undergraduate schools mentioned above.
Other schools worth mentioning : top state schools

Not too far off from our everyday knowledge of universities, yes? And very few exceptions to this combined rankings, except NYU and LSE.

I’d say you should choose a school from this ranking either for graduate schools or undergraduate schools, but look at the research ranking first for graduate school attendance, and then revealed preference for undergraduate attendance, but then don’t put too much emphasis on research quality as an undergraduate student, since you won’t get much of it at all anyway.