The Correlation between PA Scoring and Political Voting Patterns

hawkette · October 20, 2007, 9:30am

What is the meaning behind the correlation of PA scores to the political voting patterns of a state or a region? Are PA scores truly reflective of the current state of colleges in America today or are they a historical artifact/order/relative strength standing that was born in a different era, has been shaped by political perspectives and has not evolved with nearly every other facet of American life?

I looked at the top 10 private schools in Blue State America and compared them to the Top 10 in Red State America and found that colleges in blue (Democratic) states have (much) higher average PA scores than do comparable colleges in red (Republican) states. A comparison of PA scores reveals a whopping difference in average PA score of 4.75 (Blue) to 3.80 (Red). Even slimming down the comparison to the Top 5 schools, the difference was still quite large (4.88 to 4.10).

The patterns were less dramatic when comparing public universities, but still consistently favoring Blue States over Red States for Top 10 (4.06 to 3.85) and Top 5 (4.32 to 4.06) comparisons.

In a second and probably more damning evaluation, I looked at 40 colleges (top 10 privates in Blue and Red states and Top 10 publics in Blue and Red states) and compared each college’s USNWR rank to its PA rank.

For the 10 private colleges in Blue States, 8 scored higher on PA than on their final rank (Yale and Caltech were the exceptions with each narrowly outperforming in total rank over PA rank). There was a completely different pattern for Red State colleges as 9 out of their Top 10 had lower PA ranks than their final ranks (Miami of Ohio was the exception and ranked 67th in both surveys).

As we all know, the public universities do well in the PA surveys and 19 of the Top 20 scored better on PA than on their final ranks (ironically, W&M, historically ranked by academics as the premier state university for undergraduate classroom teaching, was the only exception).

Here are the numbers and comparisons:

PRIVATE NATIONAL UNIVERSITIES

PA score, PA rank, School (USN Rank), PA rank relative to USN rank

BLUE STATE PRIVATE UNIVERSITIES
4.9, PA Rank 1st, Princeton (USNWR rank 1st) , PA Rank Equal to USNWR rank
4.9, 1, Harvard (2) , Better
4.9, 1, Stanford (4) , Better
4.9, 1, MIT (7) , Better
4.8, 5, Yale (3) , Worse
4.7, 7, Caltech (5) , Worse
4.6, 8, Columbia (USNWR rank 9) , Better
4.6, 8, U Chicago (9) , Better
4.6, 8, Cornell (12) , Better
4.6 8, Johns Hopkins (14) , Better
4.75, -, AVERAGE

RED STATE PRIVATE UNIVERSITIES 
4.4, PA Rank 14th, Duke (USNWR rank 8th) , PA Rank Worse than USNWR rank
4.1, 22, Wash U (12) , Worse
4, 25, Rice (17) , Worse
4, 25, Emory (17) , Worse
4, 25, Vanderbilt (19) , Worse
3.9, 32, Notre Dame (19) , Worse
3.5, 52, Wake Forest (30) , Worse
3.5, 52, Case Western (41) , Worse
3.3, 67, Tulane (50) , Worse
3.3, 67, Miami U (OH) (67) , Equal
3.8, -, AVERAGE

BLUE STATE PUBLIC UNIVERSITIES 
4.8, PA Rank 5th, UC Berkeley (USNWR rank 21st) , PA Rank Better than USNWR rank
4.5, 12, U Michigan (25) , Better
4.2, 19, UCLA (25) , Better
4.1, 22, U Wisconsin (38) , Better
4, 25, U Illinois (38) , Better
3.9, 32, U Washington (42) , Better
3.8, 34, UCSD (38) , Better
3.8, 34, UC Davis (42) , Better
3.8, 34, Penn State (48) , Better
3.7, 39, U Minnesota (71) , Better
4.06, -, AVERAGE

RED STATE PUBLIC UNIVERSITIES 
4.3, PA Rank 16th, U Virginia (USNWR rank 23rd) , PA Rank Better than USNWR rank
4.2, 19, U North Carolina (28) , Better
4.1, 22, U Texas (44) , Better
4, 25, Georgia Tech (35) , Better
3.7, 39, W&M (33) , Worse
3.7, 39, Indiana U (75) , Better
3.7, 39, Ohio State (57) , Better
3.6, 43, U Florida (49) , Better
3.6, 43, Texas A&M (62) , Better
3.6, 43, U Arizona (96) , Better
3.85, -, AVERAGE

Now, before the attacks start cascading in, actually take the time to look at the numbers and consider these from the perspective of someone located in Red State America (btw, the effect continues as you go down the rankings). People in those Red States know that the historical powers don’t have a monopoly on brains or on academic excellence, yet the PA scores (absolutely or relatively) reflect little of this. For me, I consider the Blue State America colleges as outstanding with long histories. I also consider those mentioned for Red State America as outstanding colleges. This is not about dragging the great historical powers down; it is about recognizing others at a higher and IMO much deserved position.

When one looks at the actual data, one can’t help but see sharp PA grading disparities that (IMO) are not reflective of the absolute or relative quality of the colleges in Red State America. Instead, the PA scores reflect an educational hierarchy that was established long ago, may have been influenced/driven by political perspectives/stereotypes and has shown itself nearly impervious to change over time, either absolutely or relatively. The unfortunate result is a nearly permanent embedded discount and current underappreciation of the value of great colleges all across Red State America.

Swansong · October 20, 2007, 9:46am

It reflects a bias of blue state academics and a sense of noblesse oblige and/or protection of their traditional positions in academia. Call it a clique. They all vote for each higher with high PA scores.

It really does not matter to me. I pay little or no attention to such nonsense.

Swansong · October 20, 2007, 9:47am

I meant they all vote for each other with high PA scores.

barrons · October 20, 2007, 1:01pm

Red and Blue swing back and forth in many of those states. Wisconsin, Penn, Michigan, Minn and others have all elected Republican governors in the recent past and voted for Republicans for president prior to the GW Bush years.

kluge · October 20, 2007, 2:43pm

…and California, home of 6 of your 20 “blue state” Universities, has a Republican governor, as it has for most of the past 40 years…

Honestly, Hawkette and Swansong - this has gone past the point of diligent inquiry and entered the realm of “whining.”

mj93 · October 20, 2007, 3:00pm

east coasts states tend to be more democrat … and they house most of america’s top universities. it could just be a coincidence ?

Tarhunt · October 20, 2007, 3:41pm

This is the danger of finding a relationship and then assuming causality. Someone else could just as easily look at the same data and come to the conclusion that red state voters are dumber than blue state voters and that this is reflected in their less-than-stellar PA results. Both conclusions assume causality and, because of that, both are unsupported.

One might just as easily note that daylight and backgound noise levels increase in tandem, and make the conclusion that noise causes daylight.

If you did a mulitple regression on this using several likely factors, I would bet you’d find that PA tracks closely with faculty publishing in respected journals, which closely relates to faculty reputation in the academic world. I would also bet that geography fades to insignificance.

I also question your methodology. You are comparing PA rank to overall rank. First of all, PA is included in overall rank, so you do not have independent variables. Secondly, the overall rank is heavily influenced by factors most PA raters wouldn’t know and many they wouldn’t care much about, so the ratings are coming from a non-standard set of criteria. Next, many of the differences are certainly insignificant. Finally, a state that is blue this year may be red next year, and vice versa. If you could demonstrate that PA changed substantially from one year to the next with political voting patterns, that would be interesting.

Hey, I can just GLANCE at the list of red state private universities and know that they are not research powerhouses. That’s not to say that they are bad at research, but they’re not leaders. W&M? A classic example. If you want to do cutting edge research in most fields, W&M is not your place. It’s a wonderful school, but not for research.

Please note that my thesis is supported by the fact that public universities tend to do better on PA than on overall rank. Classes may be large, student bodies not as accomplished, but public universities tend to have greater focus on research than many (but not all, of course) private ones.

Come back to me when you’ve done some mulitple regressions and we’ll talk ;-).

SoCal18 · October 20, 2007, 3:50pm

Miami University (OH) is a public school…

jazzymom · October 20, 2007, 4:56pm

Hawkette: I think what you got here is a dog that won’t hunt.

hawkette · October 21, 2007, 8:55am

I appreciate the feedback and particularly the constructive suggestions about methodology for analyzing this. I think that my argument was poorly presented as my frustration is less with the absolute levels of PA than with the relative levels, eg, places like Rice at 4.0 or W&M at 3.7 or Wake Forest at 3.5 when schools that I perceive as producing similar quality of graduates carry much higher PAs. Is the quality of the UNDERGRADUATE faculty at these schools really that much worse than what you find at Brown (4.4) or Columbia (4.6) or UC Berkeley (4.8)??

My interest in this and in other comments is the experience that the student will experience when he/she arrives on a college campus. To the extent that one looks at Peer Assessment scoring as a proxy for faculty quality, it would be useful to know what this is measuring and if this is even relevant to the individual student. I wish USNWR would eliminate the pretense/mystery about what it measures and explicitly tie it to faculty research and the productivity of the faculty in producing this. This is a legitimate metric (though I think many might object to a 25% weighting) and would make it a worthwhile and easily-understood number in the context of a ranking system. But, as the W&M example clearly shows us, PA is almost certainly not about classroom teaching excellence.

Re my comments about differences in PA scores assigned to colleges across America, I appreciate the challenges. I agree that there are many, many fine colleges in Blue State America. But I hope you will agree that the gap has narrowed over the last decade or two, that many colleges in Red State America have seen great changes/improvements over that time and yet the relative PA scores have shown little change. Is this survey bias? I don’t know, but this sort of reminds me of the folks who say there is no bias in the media, yet 90% of the people working in America’s newsrooms vote for the national Democratic candidate.

As for the Red and Blue voting patterns that I referenced and others responded on, I should have been clearer that I was referring to the presidential elections. Going back to 1992, the states of California, Wisconsin, Pennsylvania, Michigan and Minnesota have all gone Democratic (Blue) in each election. In 1988, California, Pennsylvania and Michigan went Red, but Wisconsin and Michigan went Blue. All 50 states were Red in the 1984 Reagan landslide except Minnesota where native son Mondale narrowly won.

DSC · October 21, 2007, 10:26am

Your argument only seems to have a basis in states that are heavily in the corner of one party. Such as Massachusetts or Texas. Michigan, where 49/51 is the standard split, seems unlikely to really gain the percieved blue benifit.

Also, what is seems to show me is that older schools, with top end grad schools benifit. The Ivies, UChicago, etc. are old, and have great grad schools. Not that the ‘red state’ group doesn’t, but the Cornell, Columbia, UChicago grad group certainly qualifies above ND, Vandy, and Emory in those catagories.

Also, D1-A vs. the rest could have the same type of correlation. And that’s all it is, correlation. NOT causation.

Swansong · October 21, 2007, 10:50am

I think in a nutshell what Hawkette means, though I do not speak for her, is that the northeastern states, historically blue states, no longer claim exclusivity in being home to the most prestigious colleges and universities in the United States. The awareness of prestigious colleges around the country has risen dramatically. And that is a good thing.

I understand that these schools have been around for a very long time, often more than 100 years. But the awareness of their prestige has risen dramatically in the last 40 years or so.

hawkette · October 21, 2007, 11:15am

DSC,
I agree with both of your points, but perhaps from a different angle. First, the states that have had the largest popular vote margins, ie, the most red or the most blue, are fairly well defined geographically. The South/Southwest/large parts of the West have been almost exclusively in the Red group since the first Reagan landslide in 1980 and the popular vote margins were rarely close in those states (Florida in 2000 being an obvious exception). And we all know how solidly blue the NE and the Far West have become in the last few presidential elections (NH is the last NE state where this is still in some doubt).

I’m not pretending that my PA analysis is exact, but rather a broad statement that Red (conservative) states house colleges that don’t carry the same strength in academic circles (normally considered to be liberal) as colleges in Blue (more liberal) states. Is there a relationship? Not sure, but PA scoring (absolute and especially relative) has me wondering. If there is a relationship, it is likely similar to the media analogy where the newsfolks may perceive themselves as unbiased, but if most of their reference points and personal interactions are with others who think the same way, then the results won’t necessarily reflect other (and maybe equally valid) perspectives.

I will add that media outlets in the South/Southwest/large parts of the West offer a similar near unanimity of (opposite) opinion. Read the papers in the South, Southwest, and much of the West and compare those to the NYT, Boston Globe, Washington Post, Baltimore Sun, LA Times, etc. The differences in the reporting and the editorial content are very striking. Are these lines reflected in perceptions of colleges among academics and reinforced by the media that they consume? I don’t think that is such a far-fetched idea and part of what I was trying to communicate/explore.

As this relates to causation and correlation, I agree that geography didn’t originally create the current spreads in PA scores. But, given different regional perspectives and how these are reinforced by regional groupthink, what do you think about how this might perpetuate the PA scoring relationships, both absolutely and relatively?

For your second thought of age and grad school prominence, I definitely think you are on to something. Many have noted this relationship. IMO, the grad school prominence is probably the larger influence and point to lower PAs for undergrad-focused Dartmouth and Brown vis-

DSC · October 21, 2007, 11:36am

W&M also is not a research/grad power house. And is public. That is a horrid mix as far as I can tell.

But you would think a school that has pumped out as many presidents as W&M would have more ‘prestige’.

jags861 · October 21, 2007, 11:40am

well if it had bumped out some presidents in the last 200 years maybe it would have some…

its easy to get all the best students when you’re the only school within a 300 mile radius and there are no trains, cars, or roads.

DSC · October 21, 2007, 12:15pm

What’s your point jags?

:)

Nonetheless, you would think that a school with such a strong history would have a benifit.

Tarhunt · October 21, 2007, 2:18pm

Hawkette, you are a very bright person, and I greatly enjoy the number crunching posts you make. I appreciate your hard work and your attention to this sort of thing speaks of the kind of intelligence that finds a problem and cannot rest until it has resolved it. I like that. That’s very, very rare and will stand you in good stead throughout your life, I’m sure (I’m assuming you’re young, but I could be wrong about that).

The thing about being as bright as you are is that there’s a tendency to want to reinvent the wheel because of your justified faith in your abilities. There is a wealth of research on research techniques and their strengths and weaknesses. Your attack on the PA would greatly benefit, I think, from some reading on the topic, but let me try to summarize a few things for you and give you some insights into what might be happening when people fill out the PA questionnaire.

First off, remember that the PA questionnaire is a five-point Likert scale. That’s very, very important. First, individuals aren’t rating schools as 4.1 or 3.8 or whatever. They’re choosing a number in which a 5 mieans “distinguished” and a 1 means “marginal.” Five point scales do not allow for much in the way fine distinction, and they are generally chosen exactly because the people constructing the questionnaire don’t want people to try to make fine distinctions when such things are not possible or useful in the context of the question at hand.

In a progressive five-point scale like this one (as opposed to a balanced or centered fulcrum scale), the midpoint is usually perceived by the respondent as “average,” allowing two levels of distinction above and below average. Before you can begin to evaluate the PA, you must understand that this is all the respondent has to work with. Another key thing to understand is that, unless the points on the scale are very, very specific with very specific instructions about what each point means, progressive Likert scales are strongly subject to a rater’s perception of relativity. That is, once a rater has put certain members of a set (internal benchmarks) into one category, other members of that set will most often be distributed based on perceived qualities vs. those of the original benchmarked set.

Let me give you an example. Suppose you were asked to rate a set of human beings on a scale that runs from “very tall” to “very short,” with no instructions for where to break those categories. A rater might say to herself, “Well, Manute Bol and Shaq and Kareem Abdul Jabbar are certainly very tall, so I’ll put them in the highest category. And Danny DeVito is certainly very short, so I’ll put him in the very short category. Average would be about, oh, say 6’ tall give or take a few inches for an American, so I’ll put my husband in there. Let’s see, what about the 4 category, which would be between average height and the Manute Bol kind of height. Well, what about Kobe Bryant? That would be about right.”

Now, note that SOME people would certainly add Kobe Bryant to the 5 category, reasoning that anyone as tall as he is would certainly have to be “very tall.” So, it’s a matter of opinion, and that opinion is shaped, in this case, on very personal and, for many if not most, unstated (even internally) criteria.

Let’s move on to how someone might do a PA on various institutions of higher learning, and I’m going to use the way I might fill one of these things out as an example.

I would probably start by putting Harvard, Princeton, Yale, and Stanford in the 5 or "distinguished category. That would be fairly easy because, by just about any standard imaginable, these insitutions are world-class. After that, I would have some decisions to make. Most other institutions don’t really match up to these four overall. MIT is far more limited in its offerings. Columbia and Chicago are close, but have some departments that aren’t quite as excellent as others, and the talent of the student bodies falls just short of what you’ll find at those four places.

So, I would have to make a decision about how “elite” the 5 rating would be, at this point. I think I would decide that “distinguished” means something like “widely admired and envied” within the US, and I would include MIT, CalTech, Chicago, and Columbia as 5s, but other raters would not. What about Darmouth and Brown? Well, that’s a tougher one. Both have distinguished student bodies among the best in the country. Dartmouth’s student body is about the same as Chicago’s, and Columbia’s, but Darmouth’s faculty, overall, is not in the same league. Now, I’d have to decide whether “distinguished” also equals “reputation for providing a superior education” without, necessarily, having a universally distinnguished faculty. I think I would give them 4s, personally, but it would be a tough decision. Berkeley? The student body isn’t quite up to the other schools, but the faculty is so damn good that there’s just no way not to give them a 5. Virginia? Good, but not in this league. A 4. Same for Michigan, but I’d have to give it more thought than with Virginia. Wake Forest? Tough one. Talented student body, no doubt, and I know they have a rep for small classes and the like. I’d like to give them a 3.5, but that’s not possible, so I have to make a difficult choice. Notre Dame? I haven’t had good luck with ND students who have done graduate work in my department. Gotta go with a 3, there. What about Texas? Yikes. A student body well below those of the other 4s I’ve given, and some weak departments, but I think I really have to give them a 4 for their strengths.

If I give Virginia, Vanderbilt, Emory, UCLA, Northwestern, Dartmouth, and Brown 4s, does Wake really belong in that group? Reluctantly, I write down a 3, wishing this were a 7-point Likert scale so that I could make finer distinctions.

Do you see what I’m getting at? The PA is much like the stock market. Anytime someone sells and someone else buys, one of them is wrong. One of them is going to have made a good decision and one a bad decision. The seller is betting that the price of the sale is the best he is going to do, and the buyer is betting that the seller is wrong. The market price reflects what is known (and can be known) about the company and its assumed business prospects and the guesses, prejudices, time horizons, and many other factors affecting the buyers’ and sellers’ opinions. Some of the market price reflects sophisticated analysis, and some is just guesswork and emotion (note the dot.com bubble). Some is mystical, as in what the technical analysis people do.

Yet, the market price reflects, on a mass basis, what is known. In that sense, it is efficient, though it is not always right.

I think my observation that PA tends to be closely related to faculty reputation still holds (and, BTW, faculty rep and grad department rep are probably close to inseperable). But there are exceptions. Dartmouth and Brown are two exceptions that come to mind. Obviously, both those schools benefit from their Ivy League connections and, frankly, they may deserve their high PAs because of their focus on instruction and talented student bodies. And the research there isn’t exactly chopped liver, either.

Would it be better to use measures of “faculty quality” like publication, prestigious academy membership, and the like? I doubt it. The quest for hard numbers is an admirable one, but it does not lead to the conclusion that hard numbers are always better than subjective opinion. Would it be of much use to determine the value of paintings by the number of brushstrokes or the size of the canvas? The PA allows for adjustment due to reputation outside of faculty research reputation, and I think that’s a good thing. I seriously doubt that hard measures of faculty output would turn out to be much different from the current PA, with the exceptions of Brown, Dartmouth, and the like, and I don’t like the fact that US News would lose that subjective adjustment for instructional reputation.

dstark · October 21, 2007, 3:53pm

Tarhunt, I really like your post and your thorough analysis.

Sometimes a buyer and a seller of a stock are on the same side due to derivatives. I can’t help myself. It’s not important anyway. :)

Your analysis also holds up well when discussing SAT scores and admissions. How important are SAT scores in admissions? Sometimes we read common data sets of schools and we see choices like most important, more important, important, etc. One school may decide SAT scores are most important and another might too. Both schools may still look at scores very differently.

Many things can not be measured precisely.

That’s ok too. What one person likes another may not.

You should rate Michigan higher. :)

Anyway, great post.

hawkette · October 21, 2007, 3:58pm

tarhunt,

As usual, you make terrific sense. I really think your students are darn fortunate to have you and your straightforward (and considerate) way of explaining things.

2) I know a little about questionnaires, polling and Likert scales. Your explanations add to and reinforce my understanding and how the PA is potentially a flawed measure. Undoubtedly, your explanation does a much better job of revealing the limitations than I have.

3) On another thread, a few of us were arguing about PA scores and created our own PA scores (I call them Personal Assessment scores). Mine is reproduced below. I used a much tighter grading system of differences of 0.1 rather than 1. My personal view is that this is much more responsive to the relative differences among undergraduate colleges than the rather blunt PA instrument used by PA. Maybe USNWR should go to a finer measurement that will allow some of these gaps that I and others complain about to be narrowed to more accurately reflect the relative standing of many colleges.

4) Not all of your explanation works for me, eg, the stock market analogy. One major criticism that I have with the subjective PA (unlike the other objectively measured data of the USWNR survey) is that it is (at best) a lagging indicator of faculty reputation (not to mention the matter of how the reputation is determined differently by different people). I don’t think that the PA reflects what is CURRENTLY known.

5) As you can see from my CC history, I enjoy discussing top colleges, the students who attend these colleges, and college rankings. Sometimes my threads are based on a relatively small idea, eg, a potential relationship between PA scores and political voting patterns. My hope in most threads is to foster discussion and build on my knowledge in the process. As a result, I’ve learned a lot about what these colleges really have to offer (and don’t) for an undergraduate experience. Hopefully, others have learned a little as well and can better separate the hype from the reality as a result of these threads.

6) On the matter of my age, I’m certainly not as young as I used to be. Let’s just say that I have seen hemlines go up and down over a few fashion cycles and that I subscribe to the adage that “you’re only as young as you feel.”

(the colleges in each rank are not in any particular order)

Personal Assessment Score, Undergraduate College
5.0 Harvard
5.0 Yale
5.0 Princeton
5.0 Stanford
5.0 MIT

4.9 Duke
4.9 Caltech

4.8 U Chicago
4.8 Columbia
4.8 Dartmouth
4.8 Rice
4.8 Wash U
4.8 U Penn

4.7 Northwestern
4.7 Emory
4.7 Georgetown
4.7 Vanderbilt
4.7 Brown
4.7 Johns Hopkins
4.7 Cornell

4.6 UC Berkeley
4.6 U Virginia
4.6 Tufts
4.6 Carnegie Mellon
4.6 W&M
4.6 Notre Dame
4.6 Wake Forest
4.6 USC

4.5 Georgia Tech
4.5 U North Carolina
4.5 U Michigan
4.5 UCLA
4.5 U Wisconsin
4.5 Boston Coll
4.5 NYU

I didn’t include the LACs but should have and I think that nearly all of the USNWR top 20 LACs would score at the 4.5 level or above for undergraduate.

Tarhunt · October 21, 2007, 4:15pm

I will take your word for that, since I know exactly nothing about derivatives. ;-)

As for Michigan, THIS is why we get tenths of a point from a five point scale … because people disagree.

The Correlation between PA Scoring and Political Voting Patterns

CONNECT WITH US