Welcome to College Confidential!

The leading college-bound community on the web

Sign Up For Free

Join for FREE, and start talking with other members, weighing in on community discussions, and more.

Also, by registering and logging in you'll see fewer ads and pesky welcome messages (like this one!)

As a CC member, you can:

  • Reply to threads, and start your own.
  • Post reviews of your campus visits.
  • Find hundreds of pages of informative articles.
  • Search from over 3 million scholarships.

An critique of US News Ranking Methodology

JP_OmnipotenceJP_Omnipotence Registered User Posts: 465 Member
edited April 2009 in College Search & Selection
Ah US News. Providers of the most widely consulted ranking of colleges in America. Something I can't understand really. If you take a look at their methodology, you'll see that more of it is absolutely ridiculous. They put a lot of weight on factors that are completely irrelevant, and don't put enough on the factors that really matter, or ignore these factors entirely.

Now then, before I critique these rankings, I'd like to make a few basic assumptions about what contributes to the quality of a college. What do people want out of college? In no particular order, i'd consider the following to be what people want most out of college.

Adequate preparation for future endeavors (be it graduate school or employment)
Return on investment (salary of graduates)
An enjoyable, enriching experience.

With that in mind, let's take a look at what US News thinks is important!

Peer assessment (weighting: 25 percent). The U.S. News ranking formula gives greatest weight to the opinions of those in a position to judge a school's undergraduate academic excellence. The peer assessment survey allows the top academics we consult—presidents, provosts, and deans of admissions—to account for intangibles such as faculty dedication to teaching. Each individual is asked to rate peer schools' academic programs on a scale from 1 (marginal) to 5 (distinguished). Those who don't know enough about a school to evaluate it fairly are asked to mark "don't know." Synovate, an opinion-research firm based near Chicago, in spring 2008 collected the data; of the 4,272 people who were sent questionnaires, 46 percent responded.
What? Did I misread something? The number one reflector of a school's quality...is what other universities think of it. I'm not sure how they determined that the opinions of "presidents, provosts, and deans of admission" matter in the slightest. Why should students care what the president of Michigan State thinks of UC Santa Cruz? Why does it matter if the dean of undergraduate admissions thinks of a school halfway across the country? This shouldn't be 25% of a college's quality, it should be 0%!

If they really wanted the meaningful data, they should have consulted recruiters and graduate school departments. You know, the people whose opinions are actually going to affect the college graduates.

Okay, so we've learned that a full quarter of US News' assessment is absolutely irrelevant. Let's move on.

Retention (20 percent in national universities and liberal arts colleges and 25 percent in master's and baccalaureate colleges). The higher the proportion of freshmen who return to campus the following year and eventually graduate, the better a school is apt to be at offering the classes and services students need to succeed. This measure has two components: six-year graduation rate (80 percent of the retention score) and freshman retention rate (20 percent). The graduation rate indicates the average proportion of a graduating class who earn a degree in six years or less; we consider freshman classes that started from 1998 through 2001. Freshman retention indicates the average proportion of freshmen entering from 2003 through 2006 who returned the following fall.
I don't have a huge problem with this. Retention is important. A college that entices people to return the following year is a good one. I do have an issue with it being a full 1/5 of a college's quality though, but i'll deal with that later.
Faculty resources (20 percent). Research shows that the more satisfied students are about their contact with professors, the more they will learn and the more likely it is they will graduate. We use six factors from the 2007-08 academic year to assess a school's commitment to instruction. Class size has two components: the proportion of classes with fewer than 20 students (30 percent of the faculty resources score) and the proportion with 50 or more students (10 percent of the score). In our model, a school benefits more for having a large proportion of classes with fewer than 20 students and a small proportion of large classes. Faculty salary (35 percent) is the average faculty pay, plus benefits, during the 2006-07 and 2007-08 academic years, adjusted for regional differences in the cost of living (using indexes from the consulting firm Runzheimer International). We also weigh the proportion of professors with the highest degree in their fields (15 percent), the student-faculty ratio (5 percent), and the proportion of faculty who are full time (5 percent).
Uhh............uhhhhh.........what? Okay, i'll buy the part about the correlation between contact with professors and learning/graduation rate, but didn't we already take graduation rate into account previously??? Furthermore, what does faculty salary have to do with anything?? It's not like i'll get more out of my class just because my professor drives a Ferrari and buys his wife expensive jewelry. I agree that learned professors are nice, and it's good to have smaller classes, but why is salary more important than any other individual factor in this category??
Student selectivity (15 percent). A school's academic atmosphere is determined in part by the abilities and ambitions of the student body. We therefore factor in test scores of enrollees on the Critical Reading and Math portions of the SAT or Composite ACT score (50 percent of the selectivity score); the proportion of enrolled freshmen (for all national universities and liberal arts colleges) who graduated in the top 10 percent of their high school classes or (for institutions in the universities-master's and baccalaureate colleges) the top 25 percent (40 percent); and the acceptance rate, or the ratio of students admitted to applicants (10 percent). The data are for the fall 2007 entering class. Whether the SAT or ACT was used in making these calculations was determined by which score was submitted most often at that school for fall 2007 admissions.
Oh boy. Student selectivity. In my opinion, the least important element of a college. Even less so than peer evaluations, which received a 0%??? Of course. In my opinion, this deserves a negative weighting. Think about it. If a college is more selective, it means their students are already on the track to success. They'd probably have been successful no matter where they went to college. It's a much bigger deal if a college takes an underachiever and turns their life around, making them successful.
But hey, if we ignore this, then HYP would no longer be the top 3.
Financial resources (10 percent). Generous per-student spending indicates that a college can offer a wide variety of programs and services. U.S. News measures financial resources by using the average spending per student on instruction, research, student services, and related educational expenditures in the 2006 and 2007 fiscal years. Spending on sports, dorms, and hospitals doesn't count, only the part of a school's budget that goes toward educating students.
Yes, yes, yes. I'm shocked that this is only 10% of a college's quality. The resources a college can provide to its students should be immensely important. Yet for some reason, this is one of the least important categories in the minds of US News.

Graduation rate performance (5 percent; only in national universities and liberal arts colleges). This indicator of "added value" shows the effect of the college's programs and policies on the graduation rate of students after controlling for spending and student characteristics such as the proportion receiving Pell grants and test scores. We measure the difference between a school's six-year graduation rate for the class that entered in 2001 and the rate we predicted for the class. If the actual graduation rate is higher than the predicted rate, the college is enhancing achievement.
Okay, this is like, the 3rd time we've taken graduation rate into account. I get it, it's important, but why can't we just put it all in one category. In this case, US news is measuring its error in predictions that are based on grants and test scores...which makes absolutely no sense. I just don't understand why this is at all important.
Alumni giving rate (5 percent). This reflects the average percentage of living alumni with bachelor's degrees who gave to their school during 2005-06 and 2006-07, which is an indirect measure of student satisfaction.
Alumni giving rate: is a good way to measure student satisfaction as well as to a lesser extent their financial success. Apparently US News thinks that student satisfaction worth less than having professors that make a lot of money.
Yeah, you guys have your priorities straight.

Now then, for some factors that are incredibly important to one's choice in college but completely ignored.

Employment rate and rate at which students attend grad school: The 2 things people want out of a college education, jobs and a pathway to further education, are apparently of absolutely no importance. If your college graduates 95% of its class, but 85% of its graduates are unemployed, US News thinks of it as an exemplary institution. This is absolutely ridiculous. I cannot believe that these rankings are so popular.

Average salary of graduates: It's no secret that people go to college in order to increase their earnings potential. The salary of graduates should be another very important factor when judging a college, yet this is ignored. Shame on you US News, shame.

Tuition, housing, and financial aid: If my university experience was only 97% as enjoyable as yours but I spent 1/3 of what you spent...yeah, you get the idea. The average cost of attending the college should definitely be taken into account. Give public institutions some credit. To take into account that some schools have more generous financial aid policies, subtract the financial aid values from the amount of money students pay to attend. Of course, this isn't worth more than the above two factors, but it definitely deserves more attention than, say student selectivity.

So why do all of you pay attention to US News' college rankings when it only takes a little bit of research to see that they are...well...wrong?
Post edited by JP_Omnipotence on

Replies to: An critique of US News Ranking Methodology

  • xStevenxSteven Registered User Posts: 1,400 Senior Member
    I disagree. Student selectivity is an extremely important factor in showing quality.
  • JP_OmnipotenceJP_Omnipotence Registered User Posts: 465 Member
    Care to clarify?
  • xStevenxSteven Registered User Posts: 1,400 Senior Member
    SAT scores, while low-scorers may disagree, usually do show the intellectual capabilities of a student. The more selective schools take students with higher scores and are more selective; they admit students with intellectual promise. Students at Princeton probably display more intelligence that those at a bottom tier state school. I see what you're saying about the high scorers being able to succeed anywhere, but there has to be some measure of the overall intelligence of a student body.
  • JP_OmnipotenceJP_Omnipotence Registered User Posts: 465 Member
    Yes, SAT scores are a reflection of the intelligence of the student body, but why should this be taken into account when ranking colleges? Smarter peers means more competition for limited funding if you want to do research, more competition to gain the favor of professors. Having smarter peers won't help you find a job, won't help you get into grad school, and won't increase your salary. Sure, it can be nice to be around them, and it is nice to have intellectual conversations and discussions, but these are intangibles and should not be factored into the quality of a college. I was exaggerating when i said scores should be weighted negatively, but if I were to create a ranking system, I simply would not consider selectivity.

    Think of it this way. If the graduates of two schools are 90% employed with an average salary of $50,000 a year, but one school's student body was more intelligent to begin with, which school better prepared its students? An applicant would have a better chance of finding a high paying job if they attended the less selective school, as they would be near the top of the pecking order.
  • principalviolaprincipalviola - Posts: 2,418 Senior Member
    Curious, if you ranked the colleges which would be first? Because even with your standards I think the top would be the same.
  • modestmelodymodestmelody Registered User Posts: 4,654 Senior Member
    You make as many assumptions which are poor and unfounded as USNWR.

    Nothing to see here, move along.
This discussion has been closed.