WSJ & Forbes Rankings

nrtlax33
Thu, 13 Sep 2018 21:37:32 +0000

@Mastadon : WSJ/THE methodology states

Times Higher Education has captured student engagement across the US through its US Student Survey, carried out in partnership with two leading market research providers. For 2017 and 2018, we gathered the views of almost 200,000 current college and university students on a range of issues relating directly to their experience at college…
The survey was conducted online and respondents were recruited by research firm Streetbees using social media, facilitated, in part, by student representatives at individual schools. We also worked with participating institutions who distributed the survey to random samples of their own students.

Only one company’s name is mentioned above – Streetbees. Streetbees counts Unilever, Vodafone, L’Oréal, Pepsi, BBC as its clients. They are specializing in commercial market research, unlike non-profit/academic institutions. I think using a student representative to facilitate the recruiting of respondents on social media is pretty normal. They are doing business with some of the biggest companies in the world. I think there must be protocols/SOPs/etc. to control the randomness of their samples.

Mastadon:

The “outcomes” metric in the WSJ/THE ranking is highly suspect because:

1. It includes the a reputation survey of people in academia - which is neither objective nor a true outcome. It mimics the controversial USNews “peer survey” and is one reason for the relatively strong correlation between the two rankings.

Let’s not worry about if reputation (10%) should be categorized as part of outcomes. But they are focusing on undergraduate’s teaching (like USNWR’s best undergraduate teaching)

This pillar also looks at the overall academic reputation of the college (10%), based on THE’s annual Academic Reputation Survey, a survey of leading scholars that helps us determine which institutions have the best reputation for excellence in teaching. We used the total teaching votes from our 2017 and 2018 reputation surveys.

Mastadon:

The “resources” metric in the WSJ/THE is highly suspect … Research papers per faculty (8%)

I agree with you that undergrad focused schools like LACs, Brown will be disadvantaged by this metric since their faculty members are not focusing on graduate research to publish papers. I also agree with you that the student faculty ratio is not a good proxy for undergraduate focused resources because professors at research focused universities usually don’t teach undergraduates so they can’t be counted as resources.

An interesting point to note is that the results of the survey of academics and the survey of students (i.e. the engagement survey) have a strong inverse correlation for top ranked schools and that the schools that @prezbucky thinks are “undergraduate focused” (Princeton and Caltech) have some of the lowest engagement scores.

I kind of think of the engagement index as “pressure cooker temperature” index. If undergrads are not the focus of the campus and being neglected (very large number of graduate students) or if the school is very rigorous (like UChicago, Princeton, Caltech), the schools are likely to have low engagement score.
Read http://talk.collegeconfidential.com/princeton-university/1497933-the-daily-princetonian-survey-finds-almost-half-of-students-report-feeling-depressed.html
Why does any school want to have final exams after Christmas like Princeton? It ruins the whole holiday season.

It looks like that finally people start to read the methodologies of the rankings, which is a great thing.