WSJ & Forbes Rankings

Mastadon
Thu, 13 Sep 2018 19:38:12 +0000

The “engagement” metric in the WSJ/THE ranking is highly suspect given their description of their methodology.

It appears that they did not use the same sampling method across all schools, and one of the methods they used (using a student representative to facilitate the recruiting of respondents) is far from random and can introduce a significant bias in the sample.

Times Higher Education has captured student engagement across the US through its US Student Survey, carried out in partnership with two leading market research providers. For 2017 and 2018, we gathered the views of almost 200,000 current college and university students on a range of issues relating directly to their experience at college.

Students answer 12 core questions about their experience that are either multiple choice or on a scale from zero to 10, and also provide background information about themselves. The survey was conducted online and respondents were recruited by research firm Streetbees using social media, facilitated, in part, by student representatives at individual schools. We also worked with participating institutions who distributed the survey to random samples of their own students. Respondents were verified as students of their reported college using their email address. We used an aggregated group of respondents from both years (2017 and 2018 surveys). At least 50 responses in the 2018 survey were required for a university to be included.

The “outcomes” metric in the WSJ/THE ranking is highly suspect because:

1. It includes the a reputation survey of people in academia - which is neither objective nor a true outcome. It mimics the controversial USNews "peer survey" and is one reason for the relatively strong correlation between the two rankings.

An interesting point to note is that the results of the survey of academics and the survey of students (i.e. the engagement survey) have a strong inverse correlation for top ranked schools and that the schools that @prezbucky thinks are “undergraduate focused” (Princeton and Caltech) have some of the lowest engagement scores.

1. The Scorecard database (which is used for salaries) is based on federal student loan data which needs to be used with caution, particularly for "no loan" schools such as Harvard.

2. The "value add" statistical method they use to attempt to eliminate bias just ends up introducing a number of other biases that are less transparent to the consumer, and therefore more misleading.

The “resources” metric in the WSJ/THE is highly suspect because:

1. The student faculty ratio is not a good proxy for undergraduate focused resources because professors at research focused universities carry a lower teaching load so they can have more time for research.

For example, Caltech has a 3:1 student faculty ratio, but 9.6% of its classes have over 50 students. Princeton has a student faculty ratio of 5:1, but 10.1% of its classes have over 50 students. Clark University has a student faculty ratio of 10:1, but only 4.7% of its classes have over 50 students. WPI has a 13:1 student to faculty ratio and only 10.7% of its classes have over 50 students.

2.The number of “scholarly” research papers published per faculty member is a metric that rewards quantity rather than quality of research (which would include some notion of citations) and is likely to be inversely correlated with the amount of time a professor spends with undergrads.

1. The data that academic spending is based on does not always separate out research expenses from other academic expenses (it depends on how each school tracks its expenses) so it is neither self consistent nor representative of undergraduate spending. This mimics the USNews resource metrics and is another reason for the relatively strong correlation between the two rankings.

As an example,
Note that Caltech is amortizing the cost of the Jet Propulsion Laboratory (a major government funded research lab) over a relatively small student population - hence the appearance that they are “inefficient” in the @Greymeer calculation. Of course this also raises their WSJ and USNews rank, despite the fact that it could be argued that professors spending time at JPL are not spending time with undergrads (which would be consistent with Caltech’s low engagement score). For those who were not around in the 90’s - there was one year that Caltech was ranked number one by US News by a significant margin- because they removed the infamous “logarithmic adjuster” that was being applied to academic spending. The adjuster was re-instituted the following year, so that an Ivy League school could be returned to its rightful place at the top of the ranking.

So, the WSJ, USNews and Prezbucky’s rankings actually suffer from many of the same flaws, as well as their own unique flaws, but I would say that Presbucky’s is clearly the weakest of them all - because he likes to shamelessly promote it and I like to tease him about it