Behind the US News Rankings

sallyawp · April 11, 2008, 11:22am

I apologize for the very long post, but the following are some exerpts from an article on pressures on law schools, professors and students that result from the US News rankings of law schools from this month’s ABA Journal:

Everyone is well aware of U.S. News [rankings] and their effect in the real world, says Frank Wu, who is stepping down this year as Wayne State University Law School dean. He describes the weeks just prior to the new rankings issue as excruciating. There is always a sense of anxiety, foreboding, even dread.

It would be hard to overestimate the effect law school rankings have on everyday campus life, says Paul Caron, dean of faculty at the University of Cincinnati College of Law and a vocal critic of the magazines methodology. Students worry about how a ranking will affect their chances of future employment. Faculty worry about whether a drop in rankings will make it more difficult to be published in law reviews. Deans feel that they will be judged by how many places up or down the school moves in the rankings.

Perhaps not surprisingly, behind all this angst is an equally broad-based resentment. Since it began the rankings in 1987, the magazine is often attacked as wielding too much power; its methodology is denounced as easily manipulated and too subjective to carry such inordinate weight.

No one understands this more than Robert Morse, the man who created the law school rankings for U.S. News. As the magazines data research director, Morse says he, too, feels a high level of anxiety each year when the law school rankings are revealed.

Its very nerve-racking. I always lose a lot of weight then, Morse says. Its definitely anxiety-driven supermetabolism, because I could drink five milkshakes a day and still lose the weight and I cant do that any other time of the year.

Were the only mass media organization doing the comprehensive rankings and I feel the responsibility, Morse says.

He also feels the heat from those who resent their enduring influence. For a ratings czar, he is a very re*luctant despot. Far from being impervious to complaint, he maintains a blog where he explains his rankings and encourages constructive criticism. Hes been known to show up unannounced at gatherings likely to denounce him.

Critics of U.S. News have long called for alternatives. Some have been suggested and created, but with little popular success. In 1998, law school deans even implored prospective students to pay no attention to rankings at all.

Deans from 164 accredited U.S. law schools (in*cluding highly ranked Columbia, Cornell, New York University, UCLA, Virginia and Yale) signed a letter sent to 93,000 potential applicants saying the rankings are flawed and urging them to rely on their own observations and individual investigation.

Rankings generate huge hype, which is far more like*ly to serve the publishers purpose than the readers, the letter read.

Ten years later, the U.S. News rankings are still the guide most applicants use when pondering law schools to attend. And since the rankings are impossible to ignore, some law school deans, even longtime critics, believe it is time to parley with U.S. News and try to make them fairer.

I think rankings need to be changed, and the only way that will happen is if law school deans sit down with Bob Morse for honest discussion, Rapoport says. I would attend a meeting like that without hesitation.

Morse says he understands and agrees that the rankings are not perfect, and he would like nothing more than to discuss with law school deans ways to improve them.

U.S. News law school rankings have gone through a series of changes since 15 top law schools were ranked in 1990. Last year, the magazine ranked 100 of 184 ABA-accredited institutions as Top Law Schools for 2008, combining what used to be the top two tiers of a four-tier listing.

Specifically, according to Morse, the schools are rated on the basis of about 18 factors that fall into four general categories: selectivity, placement success, resources and reputation. The top school is given a score of 100 percent, with descending scores for all others.

Below the top 100, other law schools are tiered but not ranked. Tier III holds schools whose scores fall between the 45th percentile and the 26th percentile. Tier IV holds those that score in the bottom quarter. In both tiers, the schools are listed in alphabetical order.

Critics of the U.S. News rankings say the magazine exercises too little control over the quality of the information submitted; several of the self-reporting factors utilized in the methodology, they say, actually reward those law schools willing to cheat.

Selectivity, for instance, includes the median student LSAT scores, undergraduate GPAs and acceptance rate, and it accounts for 20 percent of the overall rating. The Association of American Law Schools has reported that some schools increase rejection ratesand boost selectivity scoresby encouraging students with no chance of admission to apply.

Placement success, which accounts for 25 percent of the overall rating, combines bar passage rates with self-reported job placement ratesstudent employment at time of graduation and those employed within nine months of graduation.

University of Texas professor Brian Leiter, who compiles his own independent law school rankings in competition with U.S. News, calls the job placement data essentially fiction.

It may have elements of truth, but basically its a work of the imagination, Leiter says on his blog.

Rapoport says reporting students as employed if they have any kind of jobwhether at a federal courthouse or a fast-food cash registeris commonplace, but she refused to do so.

There are deans who will hire a student to [photocopy] papers, work that has nothing to do with the law, and they count those students as successfully placed, Rapoport says. My school was punished with a lower ranking because I wouldnt fudge placement figures.

Resources is a category that counts for 15 percent of a schools score and includes such statistics as school expenditures per student, financial aid, number of library volumes and support services.

Still, deans complain that schools turn in slipshod or misleading stats to lift their resource ranking. In 2005, for instance, the New York Times revealed that the University of Illinois College of Law in Urbana-Champaign reported $8.78 million spent for LexisNexis and Westlaw database subscriptions, about 80 times what was actually charged. The university had, in effect, inflated its per-student expenditures by calculating the fair market value of the services, not the deeply discounted amount the school actually forked out.

But it is the dreaded Reputation category that draws the most suspicion and ire. Weighted at an enormous 40 percent of a schools total rating, it is the one category that can make or break a law schools ranking.

To measure reputation, according to Morse, U.S. News surveys 1,300 practicing lawyers, judges, deans and hiring partners, who rate the schools on a scale of 1 (marginal) to 5 (outstanding). Respondents remain anonymous.

But deans complain there is no way to know whether the anonymous people rating their schools have ever set foot on their campus, visited their websites or know anything about any of the schools programs or faculty.

Some schools work to enhance their chances the old-fashioned way: marketing.

We easily spend $100,000 on glossy marketing materials to send to the people we think will be filling out the surveys, says Wu, the Wayne State law dean. There are schools spending more than twice that. Thats money we could be spending on faculty and students. It feels like a trap none of us can escape on our own.

Caron of the University of Cincinnati edits an online publication known as TaxProf Blog that, despite being exactly what it sounds like, often becomes a forum for venting about law school rankings.

How does a school leap 21 slots or drop 20 slots in one year? one professor asks in a typical entry. Did faculty leave en masse from one to the other? Any ranking with that much volatility seems questionable.

Caron says his own university does not publicize its place in the U.S. News rankings (57th for 2008) or wheth**er it rises or falls. He is the publisher of Brian Leiters Law School Rankings, an alternative to the newsmagazines list.

But the U.S. News rankings are the most famous and influential among potential students; there is no close second, Caron says.

Caron has studied U.S. News rankings and writ*ten law review articles arguing that they lack transparency. He finds it frustrating that any schools ranking depends so heavily on the most subjective of categoriesreputation.

There is no way of knowing whether the magazine has geographical diversity among the lawyers and judges it surveys or even how many bother to respond to the surveys, Caron explains. (A U.S. News webpage says 71 percent of university deans and law school faculty and only 29 percent of lawyers and judges responded to the reputation survey in 2008. Morse said response rates are similar for all the magazines graduate school rankings.)

We recently lowered the weight of at-grad employment and increased the weight of 9-month employment as a result of many career services offices speaking to us as a group, Morse says. We also added legal writing as a specialty as a result of faculty who teach in that field talking to us.

He says efforts to manipulate the rankings are usually unsuccessful. Though the rankings can generate campuswide pain and obsession, Morse urges deans and faculty to keep them in perspective. He does not want schools wasting money or emotion on factors beyond their control.

We think that there is no proof that doing U.S. News marketing helps, Morse says. In fact, studies have shown that U.S. News marketing does not work.

When law schools refuse to file data, for instance, the magazine pulls the statistics theyve filed in the annual ABA/LSAC Official Guide to ABA-Approved Law Schools. The guide is a collaboration between the Law School Admission Council and the ABAs Section on Legal Education and Admissions to the Bar. It presents information about 194 ABA-accredited law schools gathered via an ABA questionnaire. Data is certified as fair and accurate by the law school dean, and the guide includes admission profile data and school descriptions collected by the council.

A disclaimer says the data is not intended for use in such rankings, and no form of law school rankings are sanctioned by either group.

But Morse says schools are supposed to submit the same data to U.S. News regarding library size and student-teacher ratios that they submit to the ABA. To complain that U.S. News does not fact-check the data sufficiently is to misconstrue what the rankings are. They are simply a guide, not a statistical analysis or a journalistic enterprise.

If the rankings team spots any statistical anomalies or big changes that dont appear to make sense, Morse asks the law dean to verify the data and even explain the fluctuation. Each law school signs a data verification form.

And while Morse can empathize with complaints about how easily such data as placement figures can be fudged, he doesnt feel it is appropriate for his research*ers to define what constitutes statistics like employment.

We are asking schools to report to U.S. News what they have reported on their ABA accreditation question*naire [except for employment at graduation], Morse says. If the ABA and National Association for Law Placement think the definition should be narrower, we would use another one that they had agreed on. We are open to change.

Behind the US News Rankings

Related topics

CONNECT WITH US