CSM: College presidents plan 'U.S. News' rankings boycott

Parents Forum
361

<p><a href=“johnwesley:”>quote</a></p>

<p>And, Xiggi is right, 1% of “mush” is still mush. Siserune’s reasoning is a defense of self-fulfilling prophecy, over the long-run, in perennial ranking systems.

[/quote]
</p>

<p>I don’t think that’s what either xiggi or I said. </p>

<p>Where is the “self-fulfilling prophecy” in saying that PA aggregates data from a large number of factors not otherwise accounted for by US News, or that including PA stabilizes the rankings? For instance, the selectivity rankings are oversimplified, they use the raw acceptance rate (bad for self-selected schools like Caltech or a women’s college), while the perception of selectivity that shows up in the PA ratings will be closer to reality. Now, I personally would favor fixing this by measuring selectivity better with objective factors, but as things currently stand, PA is stabilizing the rankings. </p>

<p>

</p>

<p>So the free advertising to million$ of customers in US News does not result in royalties to the college?</p>

362

<p>I believe the phrase that Xiggi used was “circular” in describing how the PA results are influenced by the raw data, and then the raw data catches up to the PA. Your justification, which you have just repeated, is that it “stabilizes” the ratings. In other words, it acts as an extremely conservative lagging indicator. I mean, we make these impassioned pleas for more accurate, up to date information, more transparency. Then, add the biggest lagging indicator out there? Does that even make sense?</p>

363

<p>So xiggi did not say or imply that “1 percent of mush is mush”, right? So much for that paraphrase, whatever his opinion of PA.</p>

<p>As to circularity, that is a rather vague comment from xiggi that he has not made sense of so far. </p>

<p>PA does not stabilize the rankings by just “lagging”. It DIGESTS the fluctuating numerical data over time and in relation to other factors (again, you do not respond to the point that padad and I have raised, that PA aggregates a host of factors not otherwise included in the USNews formula). For instance, if Harvard announces it is expanding its financial aid, or Princeton cancels Early Decision, this causes some jump in particular numbers, but the PA ratings can account for this in a less knee-jerk fashion in both the short and the long term.</p>

364

<p>siserune wrote:</p>

<br>

<br>

<p>Um, I’m pretty sure if you look up the definition of “self-fulfilling prophecy” you come up with something pretty close to what you just said. You seem to think it’s a good thing. o_0</p>

365

<p>I wrote in this thread, back in April, </p>

<p>

</p>

<p>So, I ask again in August, where are the suggestions from the colleges that have chosen to boycott the U.S. News peer assessment survey about how better to provide data on colleges? The colleges I know best on the boycott list hardly inspire confidence as paragons of high-quality education.</p>

366

<p><a href=“johnwesley:”>quote</a></p>

<p>I’m pretty sure if you look up the definition of “self-fulfilling prophecy” you come up with something pretty close to what you just said.

[/quote]
</p>

<p>So you can’t explain with any precision how the term “self-fulfilling prophecy” applies, right? Let’s walk through the logic and see what, if anything, is (according to you or other PA objectors) self-fulfilling and circular:</p>

<ol>
<li><p>Unless the PA reviewers are answering based on the previous year’s US News ratings in filling their questionnaires, it is an independent evaluation. Nothing circular or self-fulfilling there.</p></li>
<li><p>As an independent measurement of (much of) the same thing that the objective part of the formula tries to quantify, PA serves as a “reality check” on the rest of the formula. Including PA prevents results that diverge too wildly from reality, and including the numeric factors prevents PA intangibles from being all-important. </p></li>
<li><p>PA aggregates, in a qualitative way, both the US News numeric factors (selectivity etc) and many others not on the US News list. The result is the average, across many reviewers, of THEIR weights on those factors, not the weights assigned by US News. As such it is, very literally, a counterbalance to the possible arbitrariness or incorrectness of the weights assigned by US News staff, including the USNews weight of zero for factors excluded from the formula. The overall rating is a weighted average (in the ratio 1:3, according to US News) of the PA reviewers’ and the US News statisticians’ judgements of relative importance of different ranking factors.</p></li>
<li><p>For PA to affect the relative ranking of two schools (moving a lower ranked college above a higher one) their PA difference has to be A FACTOR OF THREE out of whack with the combination of all the other factors, again assuming that the 25:75 ratio is an accurate description of US News’ procedure. In other words, that 25:75 is the ratio of standardized beta-weights or something that reasonably captures the true relative influence of these factors. So unless you believe the reviewers are, as a group, totally out of tune with reality or that the 25 percent figure is misleading, then PA is unlikely to skew the rankings, though it does balance them.</p></li>
</ol>

<p>Do you agree with these points? If not, what is your specific objection to the above that makes PA “self-fulfilling” or “circular”?</p>

367

<p>I have no idea of what you’re trying to say. If you’re trying to rebut my assertion that the PA survey is extremely susceptible to a certain phenomenon known as “self-fulfilling prophecy” – you haven’t convinced me.</p>

368

<p>

</p>

<p>Siserune, I’m not sure what was so vague in my comment.</p>

<p>For starters, it was a direct reply to. “Most PA-free rankings correlate with US News, so it would hardly be earth-shaking to drop PA or change its weight.” and my words were, “Siserune, the correlation of the PA and the ranking is circular, as the impact on the final rankings is so important.”</p>

<p>Is there really a need to explain this further? How hard is it to see that the PA (with its 25% weight) represents a major component and, unfortunately I may add, obliterates much of the more subtle differences that originate from the remaining 75%. The result is that a higher PA yields a higher total ranking. making a correlation circular, since it is exactly that higher PA that caused the higher ranking. </p>

<p>For what it is worth, the choice of weighing the PA by 25% is not an accident. It is a deliberate choice by USNews since it does allow them to retool the final rankings to stabilize (or destabilize) at will. The fact that the so-called “data” is compiled by an outside firm is not as important as the fact that US News decides how to numerically handicap the schools. Again, the reasons where clearly expressed by Morse: the PA’s purpose is to HELP the public schools that lag their “peer” private schools in almost all other categories. Morse calls it rewarding the intangibles! </p>

<p>Regarding your views that the PA aggregates a host of factors not otherwise included in the USNews formula. I would be a lot happier with the PA if that is what it DID! All we know is that USNews sends a survey and collates the results that range from honest and complete to entirely whimsical, as many have admitted. </p>

<p>Further, since you believe that the selectivity ranking should be revised (although the acceptance rate accounts for only 10% of the selectivity and 1.5% of the total score) why not apply the same logic to the PA. Why not break down the darn 25% in five to fiteen categories and let the WORLD see how the schools stack up and WHAT is exactly measured and how. This would be best accomplished by splitting the rankings in two categories. The “new” PA renking could be presented in the same manner as the rankings are done today: a score and many neat columns that illustrate the final score. This way we would not have to debate ad nauseam if the PA measures quality of education, dedication to teaching, or simply a bunch of fuzzy factors ranging from nostalgic toughts of the surveyee to the latest performance in the NCAA championship, or simply that Aunt Irma attended the school in the fifties. </p>

<p>While I would prefer two SEPARATE rankings, I realize that this would never happen as Morse won’t relinquish his “stabilizing” tool, I’d be happy to see the breakdown of the PA and have sufficient information to be able to “back it out” myself from the overall rankings. </p>

<p>A good start would be to offer this ability to the suscribers of the online version. There is little doubt that the technology exists. All it would be needed is to be able to assign a dummy value to the PA and re-rank the schools. </p>

<p>Wouldn’t that make everyone happy?</p>

<p>PS “PA reviewers answering based on the previous year’s US News ratings in filling their questionnaires” is exactlty what I think happens. Of course, it assumes that the reviewers who sign the survey are also filling it in, and not a secretary.</p>

369

<p>

</p>

<p>Is there some statement you need clarified? Certainly there are some very clear statements that have gone unanswered.</p>

<p>

</p>

<p>I’m trying to find out what your assertion IS. So far it consists of repetition of a phrase with refusal to specify how that phrase applies or knocks down the statements you oppose, i.e. the claim that PA “stabilizes” the rankings and makes them more reliable.</p>

370

<p>siserune - I am not knocking down your statement that the PA stabilizes the rankings. If you re-read my post, I’m actually using your statement to support my position. “Stabilizing” is just a euphemism you seem to be peculiarly attached to for describing the same phenonmenon. You say, stabilizing, Xiggi and I say, “circular”.</p>

371

<p>I invited you (JW) several times by now to identify how, exactly, there is a “circular”, “self-fulfilling” or other effect that makes PA worsen the rankings rather than making them more reliable and closer to reality. Repeating the words doesn’t answer the question.</p>

372

<p>^I’ll make it easy on you. Let’s just use your own posts over the past three months:

  1. student surveys are “mush” (and by inference so is the PA):
    <a href=“http://talk.collegeconfidential.com/showthread.php?p=4065609[/url]”>http://talk.collegeconfidential.com/showthread.php?p=4065609&lt;/a&gt;&lt;/p&gt;

<p>2) The PA “DIGESTS the fluctuating numerical data over time and in relation to other factors” - post #363.</p>

<p>Your apt use of the word, <em>digests</em> suggests assimilation, sorting, or reclassification of some other baseline data. If the system in question winds up regurgitating the same data (albeit, in a slightly debased form), that, to me, suggests a certain circularity. </p>

<p>Now, you may push back and argue that the system is not in fact closed and that the regurgitation includes “outside factors” not included in the original mix. But, given USNews’ admitted occupation of the field, I’d say the ameliorative effects of so-called “other” marketplace influences are minimal and local at best.</p>

373

<p>

</p>

<p>A DIFFERENT and INDEPENDENT assimilation etc of the same data is not circular. That’s the whole point. It’s like asking several movie critics to rank the top 10 films of the year and averaging the results in some fashion; the redundancy of having multiple evaluations of the same data is not circularity (in fact, if it’s NOT redundant, there’s a problem), and it compensates for the biases of particular critics’ “weighting of factors” through the averaging. One can then take a 3:1 weighted average with box office receipts, actors’ salaries and other objective factors to produce a final ranking. That’s what US News does, and there is nothing in principle that makes such a procedure circular.</p>

374

<p>Why (in principle) do I need a critic to evaluate the top ten films of the year, if I’ve already seen them myself? o_0</p>

375

<p><a href=“xiggi:”>quote</a>
Siserune, I’m not sure what was so vague in my comment. </p>

<p>For starters, it was a direct reply to. “Most PA-free rankings correlate with US News, so it would hardly be earth-shaking to drop PA or change its weight.” and my words were, “Siserune, the correlation of the PA and the ranking is circular, as the impact on the final rankings is so important.”</p>

<p>Is there really a need to explain this further? How hard is it to see that the PA (with its 25% weight) represents a major component and, unfortunately I may add, obliterates much of the more subtle differences that originate from the remaining 75%. The result is that a higher PA yields a higher total ranking. making a correlation circular, since it is exactly that higher PA that caused the higher ranking.

[/quote]
</p>

<p>The only way it would cause circularity is if the PA were so influential as to basically be the ranking, with the other components as low-impact window dressing. </p>

<p>That can actually happen, if the other 75% factors place all the schools in a very narrow range while PA varies widely, thus constantly serving as the tiebreaker. In other words, if the 25:75 split is fallacious and the actual effect (for example, the standardized beta-weights rather than the raw linear coefficients) of PA is to count as 90 percent of the rankings. </p>

<p>Part of my point about correlation was that this is unlikely to be the case. The other factors considered by US News are for the most part well correlated with each other and with the PA, so they will tend to rank schools in a similar way. Other rating systems (at least, those with goals similar to US News) also produce comparable lists, although using slightly different factors or different weights on the shared factors. One of the CC frequent posters, I think GAdad, mentioned that even “percent of on-campus students” will reproduce the US News ranklist; just as endowment per student or other objective measures will tend to tell the same story.</p>

376

<p>^However, this being CC, some of us just want to test the circularity hypothesis because, 1) you know, just because the results might be interesting, and, 2) because intuitively some of us just think there is something to it. :p</p>

377

<p>See above. I offered technical advice for those wanting to reverse engineer the rankings and thus divest them of PA “mush”. Somebody else (Hawkette?) posted the PA rankings by themselves, and those can be compared and contrasted with the overall rankings.</p>

378

<p>Article in The Cavalier Daily opines that U Va should seize the moment and sign the President’s Letter now:</p>

<p>

</p>

<p><a href=“http://www.cavalierdaily.com/CVArticle.asp?ID=30629&pid=1607[/url]”>http://www.cavalierdaily.com/CVArticle.asp?ID=30629&pid=1607&lt;/a&gt;&lt;/p&gt;

379

<p>“Drew’s president is making us think”</p>

<p>

</p>

<p><a href=“http://www.northjersey.com/page.php?qstr=eXJpcnk3ZjczN2Y3dnFlZUVFeXk2NSZmZ2JlbDdmN3ZxZWVFRXl5NzE5NDA1NyZ5cmlyeTdmNzE3Zjd2cWVlRUV5eTQ=[/url]”>http://www.northjersey.com/page.php?qstr=eXJpcnk3ZjczN2Y3dnFlZUVFeXk2NSZmZ2JlbDdmN3ZxZWVFRXl5NzE5NDA1NyZ5cmlyeTdmNzE3Zjd2cWVlRUV5eTQ=&lt;/a&gt;&lt;/p&gt;