<p>Tarhunt, your analysis was excellent. I enjoyed reading it. I just threw the Michigan remark in there because my daughter goes there. ;)</p>
<p>Hawkette:</p>
<p>
</p>
<p>In that case, I have failed miserably. The PA is not flawed any more than a consensus among English literature professors that Shakespeare is the greatest writer in the English language is flawed. It is what it is. There’s no flaw implied. In that consensus, one may find that other writers, such as DH Lawrence, Eudora Welty, John Milton, James Joyce, or GB Shaw also get votes as the “best.” What one is measuring is opinion, and there can never be any flaws in the result other than methodological ones and well-known ones of validity. Such results can be misinterpreted, but that is not the fault of the design.</p>
<p>My intent was just the opposite. I was trying to explain why your quest for validity, in this case, it futile. No such means of proof exists, because there are no benchmarks.</p>
<p>
</p>
<p>This would not be methodologically sound, because such fine distinctions cannot be measured without very specific and very measurable criteria. I can hand my students a one-meter stick without any gradations on it, and then ask my students to tell me, within .5 mm., the length of an object. Obviously, they will be unable to do it because they are not equipped with the measurement device that will allow them to do this. That’s obvious. Unfortunately, some people DO attempt to do the impossible and only to bad effect. For instance, it’s quite common for engineers who are also managers to carry out performance evaluations for their subordinates to ten thousandths of a point. That sort of accuracy is an illusion, in the same way that manipulating numbers that have been rounded to tenths can never be more accurate by carrying them to many more decimal places.</p>
<p>I’m sure this is WHY there is a five-point Likert scale. The people designing the questionnaire decided, and rightly so, that their raters could not reasonably make finer reputational distinctions than that. I think one could have argued for a seven-point scale, but five is perfectly defensible, and may even be better than a seven-point scale, given the subject matter at hand.</p>
<p>Hawkette, what I’m trying to say is that rating something as broad as “reputation” with a 40-point scale (the distance between 1 and 5 x 10) is handing out that meter stick and demanding measurements that can’t be made. Garbage in. Garbage out.</p>
<p>
</p>
<p>A lagging indicator in this case is probably warranted, since institutions change very slowly. It’s not as if Middle Tennessee State will be able to become one of the great institutions in the world next year, and this lagging indicator will miss it. Reputation is what is is. It is the rare field in which reputations are made overnight, and the rare field in which they are ruined overnight. Reputations are generally not built or lost based on short-term factors, especially for institutions that have been around for centuries.</p>
<p>However, reputations DO reflect what is currently known on a MASS basis. Some may know more than others in individual areas, but the measurement of opinion does, in fact, measure what is currently perceived by the respondents. After all, that’s the point of opinion surveys.</p>
<p>As for reputation being determined in different ways be different people, in what walk of life is this not true? Hawkette, you are looking for the Philosopher’s Stone here, and it does not exist. Reputation is what it is. It is opinion, in the end. Your quest to boil opinion down to quantitative factors in bound to fail. You CAN identifiy many factors that influence opinion with a mulitple regression, but even then, you will not get them all because some of them are not measurable.</p>
<p>As for your ratings, I find them to be useful only for understanding your opinion, but of no value beyond that point. If you have found absolutely objecctive criteria for your 40-point scale, then I’d be interested in your sharing them, but I doubt you have. You’ve simply found a way of expressing your own opinion on a scale that is methodologically unsound. To put this another way, if you presented this scale to me on one of my classes in research technique, you would fail.</p>
<p>Hawkette:</p>
<p>A note on the last post. I should have added to the final words “you would fail” by saying that, of course you wouldn’t use that kind of scale for opinion because you would have been well-versed in research techniques by the time you submitted your scale. ;-)</p>
<p>
</p>
<p>Hawkette, why do you assume there is a separate faculty for undergraduate teaching? </p>
<p>In my undergraduate classes at Cal, I was taught by top professors who also taught their graduate researchers.</p>
<p>
</p>
<p>I think there is a good point here. There really doesn’t have to be thread after thread after thread here about a magazine that a lot of us mostly disregard in planning our college application lists. Let’s go right to the most extreme possible case: even if it was found that peer assessment scores were exactly the result of pay-offs by colleges to the magazine editors (which surely isn’t true), the harm would be minimal, because most students have local sources of information (friends, relatives, school counselors) about local colleges, and most students go to college fairly close to home. The great majority of students who matriculate at colleges neither know nor care what their college’s peer assessment score is.</p>
<p>tokenadult:</p>
<p>First off, let me thank you for the work you did on the 168 thread. It’s appreciated.</p>
<p>Personally, I find the US News rankings useful, but only in the context of what they are and not what they aren’t, and that has to do with an interpretation of the methodology. I believe you are in error about the US News impact. Even if we didn’t have the anecdotal evidence of how strongly so many colleges are reacting to US News ratings, we’d have at least one study demonstrating that falling a few places in the rankings reduces applications, yield, and increases the amount of aid a college must give to get the students it wants.</p>
<p><a href=“http://blast.mbhs.edu/college/Rankings.html[/url]”>http://blast.mbhs.edu/college/Rankings.html</a></p>
<p>So, despite studies of self-reported student behavior in which students say that US News rankings have little impact on decisions, we have actual results that suggest that, as usual, self-reported behavior on controversial issues is unreliable.</p>
<p>I don’t blame USN for this. They are very clear about their methodology and use copy to explain that their ratings aren’t the be-all and end-all for college selection. I blame those who treat the ratings that way.</p>
<p>Hi, Tarhunt, I guess how I’d relate your thoughtful comments to what I said is that I think the part of the U.S. News rankings that is indisputably influential is the part that is most based on indisputably relevant information: which college has better financial support, what the students bring to the college with their own abilities established through high school study, what the staffing ratio is at the college, etc. It was something new when U.S. News put all of the national universities in the United States on one ranked list, and all of the national liberal arts colleges on a separate ranked list, and allowed readers to see how many colleges “outranked” their best-known college. A lot of guidebook series existed before U.S. News got into that business (some have since gone out of print, because they were full of inaccuracies), and of course colleges have always had more or less widespread reputations, but U.S. News did do something new and different by rank-ordering a whole huge list of colleges. </p>
<p>I think the underlying reality is that colleges in the United States are what is called a partially ordered set. </p>
<p><a href=“http://en.wikipedia.org/wiki/Partial_order[/url]”>http://en.wikipedia.org/wiki/Partial_order</a> </p>
<p>Various publishers have come up with various attempts to define a linear extension of the whole set of colleges in the United States based on a “quality” ordering relationship, but any such attempt is debatable. I’m all for people giving specific, affirmative reasons why one or another college is a good college to apply to or to enroll at. I’ve actually been very interested to observe the different approach taken by Washington Monthly magazine </p>
<p><a href=“http://www2.washingtonmonthly.com/features/2007/0709.collegeguide.html[/url]”>http://www2.washingtonmonthly.com/features/2007/0709.collegeguide.html</a> </p>
<p>in ranking the same group of colleges that are ranked by U.S. News. In some cases, I like the Washington Monthly rankings much better–but I acknowledge they are debatable too. </p>
<p>I really don’t think most high school students care AT ALL what the peer assessment scores are for any college in the annual U.S. News issue. The academics at the disfavored schools have all learned how to decry those assessments 
so I think all the affected constituencies are already dealing with that issue. Colleges are all still free to do any illustrious thing they like to burnish their reputations.</p>
<p>Tarhunt,
Again, thanks for the lessons. Your comments make it clear that the PA is “garbage in, garbage out” and on that we can agree. </p>
<p>As for the reputations of non-academic institutions, I disagree with your view that they don’t change often. I see the fortunes and corresponding reputations of businesses wax and wane significantly in the space of a few years. Heck, look at Google….and at Lycos. Or look at the list of companies that make up a major stock index (like the S&P 500) and see how its make-up changes rather significantly over a decade or shorter. I see careers of people across all types of industries likewise flow and ebb in a short period of time. Look at Hank Greenberg….and Robert Nardelli,and Carly Fiorina. Frankly, when I look at American business and life, I see much more change than not. </p>
<p>But I never seem to see any real change in how academics rate American colleges. There seems to me to be a permanent pecking order that almost no matter what a college does (maybe even MTSU), it will not see its reputation change among academics. Students may come to love a college and the profile is rapidly improved. Businesses and employers may come to love a college and flock there to hire the graduates and see them flourish in the real world. But it appears to me that academics are unlikely to ever recognize this and the school is forever consigned to mediocre ratings. If you’re one of the lucky ones on top, this system seems just groovy, but talk to the folk that have seen sharp improvements with no corresponding rise in recognition by academics and get ready to hear an earful. These people aren’t stupid, but an unseasoned reader of college rankings could unfortunately get that impression. </p>
<p>tokenadult,
With all due respect, I think you greatly underestimate the impact that USNWR has on college searches and ultimate selection. Just last week I had correspondence with a college sophomore who posts actively on CC who made a college selection for a (highly ranked) college and her selection was heavily influenced by a difference of three ranking spots according to USNWR. I think she looks back now and cringes (as undoubtedly you and I would) at how she made her ultimate decision, but the reality is that at the time she was going through the college process, she didn’t have the perspective to realize that the rankings differential was essentially meaningless. She knows now and she’s doing just fine, but this could have resulted in an unfortunate college choice. God knows how often this is repeated, but to dismiss the rankings and their influence, even for the most intelligent of high school users, would be incorrect.</p>
<p>token:</p>
<p>I agree with the idea that colleges are a partially ordered set. In that sense, any attempt at a definitive, linear order must be doomed to failure, except, of course, when the methodology is known and, thus, the ordering can be evaluated based on the strengths and weaknesses of the methodology.</p>
<p>I was also around when the US News rankings began, and agree that they were something different, and would assert that they improved over time as the formula was revised.</p>
<p>The point I was trying to make and did not do well is that the rankings DO matter because they clearly have a material, financial impact on institutions. That’s unfortunate, but appears to be true.</p>
<p>Tarhunt:</p>
<p>Wonderful posts, thank you. However, a stock seller and buyer can both win. Perhaps the stock has increased by a lot, and a cc parent needs to cash out to pay tuition. However, the stock still has potential upside, so a buyer is interested.</p>
<p>Hawkette: Personally, I just love to log into cc to see what the next long, strange trip it’s been for PA (a play on Grateful Dead). Instead of inventing data comparisons (which really do question your credibility), why don’t you just come out and admit that you do not like PA bcos its an opinion survey and you believe that it offers no value. Simple. :)</p>
<p>Or why not simply make a positive case for the desirable characteristics of your favorite college? If there is a case to be made, let’s hear it, with specifics.</p>
<p>bluebayou:</p>
<p>I freely admit that my stock market analogy was simplistic. Of course one might sell off just to make ends meet, increase liquidity against risk, etc. I used the stock market only to show that stock prices, according to some economists, are perfectly efficient, reflecting what is known and guessed in the marketplace at any one time.</p>
<p>tokenadult:</p>
<p>I agree. I really think the best thing hawkette can do at this point is come up with his/her own ranking system and defend it.</p>
<p>Tarhunt,
Not sure how I could have misinterpreted your comments as much as I did, but apparently I did. My apologies. No intention to twist your words or take out of context: I read your words as someone who has a similarly low view of PA scoring and the methodology that it uses. I guess I got that wrong. Sorry for misinterpreting your words. </p>
<p>As for creating my own ranking, if you really are interested, Id be happy to oblige and share my own suggested model. </p>
<p>Bluebayou,
Youre right. I dont like PA, but not necessarily because it is an opinion survey. I can accept that folks in academia have an opinion and I understand that their voice is important to some students, families and others. However, I dont like PAs lack of definition, I dont like its exclusivity to academic viewpoints, and I dont like its highly static nature. Id probably be amenable to opinions if there were also independent voices that supported that opinion, eg, the other stakeholders in undergraduate education-thestudents, employers, alumni, etc. But IMO ideally either all of this opinion or none of this opinion and its subjectivity should be in the ranking. The best solution is to separately list any opinion (academics, classroom teaching, students, employers). </p>
<p>Tokenadult,
I have made known my favorite schools numerous times, but not because of PA scoring or their relative lack of good scores. PA is something that I feel strongly about, but my favorite group of colleges spans the PA spectrum. I prefer colleges that provide an undergraduate experience that is excellent inside AND outside of the classroom, where the students are both smart and know how to have a good time, where there is an active social scene, and where the athletic scene is fun, exciting, of decent size and has national consequence. As I have posted numerous times, that leads me to private colleges like Stanford, Duke, Northwestern, Rice, Vanderbilt, Notre Dame, Georgetown, USC and Wake Forest and the publics of UC Berkeley and U Virginia. After HYPM, I would choose any of these colleges because I truly believe that these colleges offer the best combined undergraduate experience (academic, social, athletic) in the country.</p>
<p>
</p>
<p>Maybe if Emory, Tulane, or USC profs win a Nobel prize like a U Chicago prof just did (again for econ, thus continuing U Chicago’s great econ reputation), academics might start to change their minds.</p>
<p>If you were going to seriously, with a straight face, explore the theory that being in a so-called “Blue” state will somehow boost a college to a higher-than-expected rating, then I’d expect you to look at all the colleges, not the top few as you have done.</p>
<p><a href=“http://skepdic.com/confirmbias.html[/url]”>http://skepdic.com/confirmbias.html</a> </p>
<p><a href=“http://www.devpsy.org/teaching/method/confirmation_bias.html[/url]”>http://www.devpsy.org/teaching/method/confirmation_bias.html</a></p>
<p>Part of Heider’s consistency theory and embedded in cognitive dissonance. Thanks token.</p>
<p>hawkette, I can only speak for myself, but at times you have come across as someone who really resents the fact that some schools have high, possibly undeserved, PAs (and therefore undeserved rankings), and that you will therefore jump to start a thread that will show those schools in a less-flattering light. Sometimes those threads seem built on fairly thin premises. Maybe this is a false impression but it’s the one that I get sometimes.</p>
<p>The other thing that I don’t understand is your tendency to take interesting topics and elevate them into some kind of moral crusade. Comments like this, for example:
</p>
<p>I mean, honestly, this is a conventional academic truth, according to you? Do you truly believe that there is a widespread belief in academia (or the public) that faculty quality is so rarely found? I find it hard to believe you subscribe to an idea that is preposterous. It’s as if you need to imbue your conversations on CC a noble purpose. </p>
<p>It’s unnecessary. </p>
<p>First, it is complete bunk that there is a belief in higher ed that a few schools have a monopoly on faculty quality. **Second, your contributions are potentially interesting and useful just as they are. **You don’t have to weave this fiction about conspiracies, or suggest that institutions like Vanderbilt & Rice are struggling for students, or that applicants are making dreadful, life-shattering mistakes when they pick colleges for semi-daft reasons. I like talking about these things, and you’ve got interesting insights that I think many of us can benefit from. But, IMO, it gets tiresome when we do so under this false sense of impending calamity.</p>
<p>To the original question:
</p>
<p>What about looking at the PA marks for schools on princetonreview’s Students Most Nostalgic for Ronald Reagan & Bill Clinton lists? For the most part, these are not top 50 NatU or LAC schools. If there are no significant differences in PA between the 20 schools on each list, then I don’t think that it is worth looking for elsewhere. Since the schools appear on a number of lists, then look at the PA score relative to the 3 (or 4) schools before and after each school. I don’t have access to current the USNWR data or I’d work up the numbers myself. (05 is the most recent USNRW PA data I have).</p>
<p>standrews:</p>
<p>Not a bad idea but, once again, one has to be very, very careful about assuming causality in any statistical relationship. There are pairs of things that are closely related in such a way that one of them can mimic “cause” without, in fact, having any (or very little) impact whatsoever.</p>
<p>Here’s my take on why the PAs in “red” states tend to be lower than those in “blue” states. I have already stated my claim that I suspect (though I can’t prove) that PA is closely related to the perception of faculty excellence in the academic community. There are exceptions. Darmouth and Brown probably have higher PAs than reputation for research excellence warrants, but that can be explained by age and Ivy League status. I believe my thesis is supported by the fact that large, research-oriented, public institutions tend to do well on PA vs. their overall US News ranking. I can think of no other legitimate reason why publics should do so well on the PA, though I admit there may be something I’m missing that only research would uncover.</p>
<p>Let’s take a look at a legitimate, alternate explanation for why red state schools tend to do poorly on PA compared to their overall US News rankings.</p>
<p>First, red states are overwhelmingly in the South, Southwest, Plains, and Rocky Mountain West. In the Midwest, the only true red state (I believe) is Indiana. Ohio is a swing state. </p>
<p>These states are traditionally, and in many cases stilll are, poor compared to blue states. Even Indiana’s industrial base was mostly along the northern lake shore and tied closely to Chicago. Most of them were even poorer at the time that government research grants to higher learning exploded during and after WWII. These research grants tended to favor schools with advanced departments in technological areas or in the hard sciences. Naturally, those states with well-developed industrial bases tended to have strong departments in these areas already in place, and benefited mightily from these grants. Their departments grew. The faculty members, having substantial resources at their disposal, began to turn out first rate research and, as they did so, their reputations grew and attracted other very bright faculty members.</p>
<p>This is the case to this day.</p>
<p>The primary beneficiaries of this government largess were the Big 10 schools, Ivy League schools with top physical sciences and mathematics departments, the University of California system, etc. The University of Texas and Texas A&M probably also benefitted. Since that time, the rest of the country has been playing catch up on the research end. Large, distinguished research departments tend to get more, bigger, and more prestigious grants because, frankly, they are more likely to actually produce useful research results. As a taxpayer, I’m not against this, but it does make playing catch up much harder for the other schools.</p>
<p>So, my guess is that the higher PAs are fairly closely related to overall faculty reputation, that faculty reputation is closlely related to research reputation, and that research reputation is closely related to being in position to capitalize on the huge influx of government money during and following WWII that has created an advantage to this day.</p>
<p>To me, a more interesting topic might be, “Why are red states and blue states so split among economic lines?”</p>