How do top scorers on tests fail to gain admission to top schools?

kluge · August 7, 2007, 1:37pm

newmassdad -
The point I’m making is about the selection of “their goal” and how that determined what they identified as a “component” - and how that got morphed into Calmom’s comment that “As I understand it, the SAT was designed and validated only to predict first year grades; there is NO (-0-) statistical correlation or predictive value of test scores after the first year.” Her comment is understandable given the spin that’s been put on these studies; it’s just that the data in the studies doesn’t seem to me to support the spin. What I see is that grades alone or tests alone are roughly equivalent in predictive value, year 1, year 4, or cumulative for 4 years (with a slight edge to test scores in each case.) A combination of grades and a test component (either SAT1’s or SATII’s) is better still - although not all that much - and the additional improvement from adding the other test component to that mix doesn’t make much additional difference (not surprisingly.) If you pick grades + SATII as your starting point, it’s true: adding SAT1’s doesn’t do much. It’s also true that if you start with Grades + SAT1 adding SATII’s doesn’t really add much either. (See second study, table 4, models 4, 5 and 7) That’s because the added predictive value of each type of test would apparently (and logically) be largely duplicative of the added value of the first set of tests.

Range restriction is also covered in fn 19 of the second report. Essentially, as I read it, given the self-selection of the applicant pool to begin with they could use a selection formula which utilizes grades only, or tests only, and have virtually no significant change in “success” as variously defined; even more significant is that they could use a random drawing among the applicants and get a “success” outcome that would not be all that different from any of the other formulas!

newmassdad · August 7, 2007, 2:27pm

kluge,

there’s a lot more to the story than even you discuss. Ideally, for example, one would like to see the cross correlation cooeficients among the various predictor variables. No doubt they are very high. One would also want to see the residual plots for the various components of their regression analysis. Nonetheless, the data they present is overwhelmingly supportive of GPA being the strongest predictor, followed by SAT II (and really the writing test within that, hence their push at the CB to add writing!).

Regarding Calmom’s earlier comment about the SAT, I don’t think the SAT was “designed” to predict first year grades. Rather, it was validated against that metric. I am also not aware of any study that shows it has NO predictive value for later years. Indeed, that would be like saying first year grades don’t matter either. (if the SAT predicts first year grades, and first year grades predict later performance, than it is probable that the SAT would also predict later grades…but not proven until studied)

Finally, you misunderstand range restriction, I think. A better way to look at range restriction is to recongize that HS GPA among matriculants is not a normally distributed variable (I’m guessing…) Rather, it is skewed to one end. This impacts the statistical power of the subsequent analysis.

Now, it IS fair to say “of those admitted, HSGPA accounts for X%” of first year grades. But, adcoms don’t use that restricted range. They look at a broader range and reject a lot…you get the idea.

Kluge, was this report before, after, or part of the decision to junk the SAT at U Cal? It might help to understand any underlying biases.

xiggi · August 7, 2007, 2:58pm

And what was this perspective? Was the purpose of the study not to massage sufficient data to support the prior admission policies at the UC? Is it unheard for a UC leader to reaching a decision based on witnessing his granddaughter academic “preparation,” making a far-reaching statement at a national conference, and THEN asking his staff to corroborate the “findings?”

Further, does the perspective of the UC reports have much validity outside the boundaries of their own population? Doesn’t the UC student population exhibit academic and SES characteristics that are very different from the rest of the country?

We find spin and spinmeisters everywhere!

kluge · August 7, 2007, 3:03pm

Newmassdad - you obviously have a much better handle on the statistical analysis than I do. The first study was part of the UC pressure on CB which resulted in the addition of the writing section to the SAT. UC didn’t dump the SAT - they just (in essence) made the old Writing SAT II part of the SAT 1. To be honest, although Xiggi and I disagree about a lot of things, I think he’s probably right on this one. I think the main function of the studies was to provide cover for policies which would allow the University to enroll more lower SES students in higher-prestige campuses without running afoul of the anti-affirmative action zealots. Hence the extensive discussion of social issues in the studies.

Since it’s already way too late to avoid beating a dead horse, let me articulate what my criticism is of the two UC studies and how they have been interpreted to the public. The basic framework that has been used to suggest that test scores have little or no predictive value has been to gloss over the fact that either grades or test scores standing alone have about the same predictive value (which edges out the other depends on whether the SES adjustment is made or not.) Grades and one type of test are combined, yielding a somewhat better predictive value, which cannot be significantly improved upon by adding more test scores into the mix. The remaining set of test scores is then declared to be irrelevant.

My thought experiment: What if grades were divided into classes, like test scores? (UC probably has the ability to do this with their admissions system and its specified “a to g” course classifications.) Simplifying it a little, what if each student had their GPA calculated separately for each of five categories - say English and Literature, History and Social Studies, Foreign Languages, Physical Science, and Math. One combined GPA, five sub-GPAs - just like the SAT’s with 3 SAT1’s and 2 SATII’s. We know from the first two UC studies that the predictive ability of the total GPA alone would be similar to the predictive ability of the total test scores alone. My hypothesis is this: If a collegiate success prediction study was performed using test scores as a single number coupled with various subject area GPA groupings - the mirror image of the studies which were performed - you’d get the same result as the studies which have been done, except that now the “irrelevant” factor would be the last subject area GPA added to the mix, not one of the tests.

Example: Predictive value of tests alone is about 20. Predictive value of grades alone is about 20. Predictive value of all grades and all tests combined is about 26. My suggestion is that the predictive value of all tests + English, History and Science subject GPAs would be — about 26, and that adding the Math and/or Foreign Language GPA’s would not make a significant difference in predictive value. If I’m right, would that mean that high school Math GPA has little or no significance in predicting college success?

As an aside: I, too find it surprising that a single 1 hour writing test has - by itself - almost as much value in predicting collegiate success as four years of grades.

Xiggi - I disagree about California being another planet from the rest of America. One out of every 8 Americans lives here. We are America, like it or not.

marite · August 7, 2007, 3:13pm

Xiggi is there, too!

xiggi · August 7, 2007, 3:17pm

Yes and no!

On the one hand, in theory there SHOULD be a high correlation between the performance of students who were able to produce a reasonably well-crafted and logical essay as well as have a competent mastery of grammar. On the other hand, the correlation becomes more interesting when one considers how the SAT Writing test was eminently exploitable as a successful SAT Tutor demonstrated by having his students MEMORIZING a perfect essay and delivering it without much attention to the … prompt!

So, in the end, a candidate at the UC could ace three SAT II tests by mastering a TI-89, memorizing an essay, and knowing enough of second- or third-grade level Chinese or Korean.

Interesting, indeed!

system · August 7, 2007, 3:18pm

Just to let you know… the # of ‘perfect scores’ is a little higher than 200-something because of superscoring.

There are many who manage to get a 2400 after taking the SAT a few times… if they aren’t impressive anywhere else… the top colleges will see no reason why that person should be accepted over many others with lower SATs.

xiggi · August 7, 2007, 3:23pm

Kluge, is THAT what I wrote? My comments, "Further, does the perspective of the UC reports have much validity outside the boundaries of their own population? Doesn’t the UC student population exhibit academic and SES characteristics that are very different from the rest of the country?" were confined to the validity of the UC reports for the REST of the US.

For instance, what would a comparative study of the UC, the University of Texas, the University of Michigan, and the University of Florida show?

Take a look at the population distribution in the various states! Take a look at the admissions’ system and weigh the different criteria. Take a look at the SES distribution … why does the UC have such a high average GPA? Why are there so many Pell grantees at Cal and UCLA and how does that correlate with the conclusion that the SAT tracks high SES factors?

Why do you assume I don’t like California and Californians? Can’t I take a critical view on the UC system or the education policies in California without disliking the State?

mammall · August 7, 2007, 3:29pm

Why do people love to dwell on annecdotal evidence that perfect SAT and ACT scorers get rejected by top schools? Me thinks these folks harbor envy deep in their begrudging hearts.

kluge · August 7, 2007, 3:29pm

Yeah, Marite - but Xiggi’s just visiting! In fact, a big factor in what got me started thinking about all this was Xiggi’s alma mater, Claremont McKenna. That’s the only selective school I know of which officially states that it doesn’t consider GPA or class rank to be among the most important factors in its admissions decision. And CMC is an interesting case study as a college. A relatively recent arrival on the academic scene, a meteoric rise in prestige (meteoric by comparison to the glacial moves of the various rankings, anyway) and an unabashedly ambitious institutional philosophy. Is there a connection there?

<edit> I meant no offense Xiggi; simply that the UC system’s applicants - as a whole, not just Berkeley and UCLA - strike me as probably being a fairly valid sampling of America’s collegiate universe. Maybe a few more Asians and Latinos, but otherwise not much different from UT, UM or UF, I wouldn’t think.</edit>

xiggi · August 7, 2007, 3:39pm

Kluge, no offense taken! None whatsoever.

I believe that you might be surprised at the differences between the various state systems.

tokenadult · August 7, 2007, 4:07pm

I see that there have been more than 100 replies to this thread, and I thank all of you for your interesting contributions to the discussion. I’d like to go back to a numerical issue that has come up in more than one reply. The College Board reports that in the high school class of 2006 there were only 238 distinct individuals who attained a single-sitting SAT Reasoning Test composite score of 2400.

<a href=“College Board - SAT, AP, College Search and Admission Tools”>College Board - SAT, AP, College Search and Admission Tools;

Similarly, ACT reports that only 216 individuals in class of 2006 attained an ACT composite score of 36.

<a href=“http://www.act.org/news/data/06/pdf/National2006.pdf[/url]”>http://www.act.org/news/data/06/pdf/National2006.pdf</a>

(table 2.1)

These scores are RARE. (There is some overlap between the two peak-scoring groups.) For perspective, consider the size of the entering classes at Harvard, Yale, Princeton, and Stanford. The figures are as found on the College Board College QuickFinder application on the Web, based on the Common Data Set methodology and definitions.

Harvard

First-time degree-seeking freshmen: 1,686

<a href=“http://apps.collegeboard.com/search/CollegeDetail.jsp?collegeId=1251&profileId=0[/url]”>http://apps.collegeboard.com/search/CollegeDetail.jsp?collegeId=1251&profileId=0</a>

Yale

First-time degree-seeking freshmen: 1,315

<a href=“http://apps.collegeboard.com/search/CollegeDetail.jsp?collegeId=4123[/url]”>http://apps.collegeboard.com/search/CollegeDetail.jsp?collegeId=4123</a>

Princeton

First-time degree-seeking freshmen: 4,760 [sc. should be 1,232, as reported on the Princeton Web site. The mistaken figure shown is total undergraduate enrollment.]

<a href=“http://apps.collegeboard.com/search/CollegeDetail.jsp?collegeId=4221[/url]”>http://apps.collegeboard.com/search/CollegeDetail.jsp?collegeId=4221</a>

cf.

<a href=“Facts & Figures”>Facts & Figures;

Stanford

First-time degree-seeking freshmen: 1,646

<a href=“http://apps.collegeboard.com/search/CollegeDetail.jsp?collegeId=3387[/url]”>http://apps.collegeboard.com/search/CollegeDetail.jsp?collegeId=3387</a>

The only highly sought-after national research university that has a small enough entering class to admit mostly peak scorers is Caltech.

Caltech

First-time degree-seeking freshmen: 214

<a href=“http://apps.collegeboard.com/search/CollegeDetail.jsp?match=true&collegeId=4214&type=qfs&word=California%20Institute%20of%20Technology[/url]”>http://apps.collegeboard.com/search/CollegeDetail.jsp?match=true&collegeId=4214&type=qfs&word=California%20Institute%20of%20Technology</a>

All other colleges have to dig deeper down into the score distribution just to fill their classes. (Well, Caltech does too, because not all peak scorers apply to or matriculate at Caltech.)

Some replies have commented that most selective colleges “superscore,” that is take a student’s best scores section-by-section if the student reports more than one SAT Reasoning Test score in the admission process. I doubt that that increases the number of 2400 scorers enough to fill a whole class at Princeton, much less at Yale, Stanford, or Harvard. I doubt this because a plurality of test-takers report just one set of SAT scores

<a href=“College Board - SAT, AP, College Search and Admission Tools”>College Board - SAT, AP, College Search and Admission Tools;

rather than two, three, four, or five sets of scores, and most of the students who do retakes are near the lower rather than the higher end of the score distribution. I’m sure there are some students who gain a “superscored” 2400 without ever attaining a single-sitting 2400, but I doubt such students even double the number of 2400 scorers available to all colleges around the country to consider for admission. (When I make a statement such as this I am of course inviting anyone with definite information on the subject to post a link to a source.)

What’s the upshot of this? On the one hand, all of the highly selective colleges have to find students with less-than-perfect scores who have other desirable characteristics to fill out their entering classes. But, and this is crucial for students to keep in mind, even though peak scores are scarce, they are not always sufficient for gaining admission to the school the peak-scoring student desires. That peak-scoring students are sometimes passed over (as has been reported by several replies in this thread) even though they are scarce demonstrates that college admission committees sometimes value some other characteristics of students even more than they value test scores. Then, for the student still in high school, the question becomes what those other characteristics are, and perhaps how to develop those characteristics while still in high school.

Thanks for your thought-provoking replies.

newmassdad · August 7, 2007, 4:20pm

Tokenadult,

There are many alternative explanations to your hypothesis:

Try this:

<ul>
<li>colleges recongize that a perfect score is likely to be a probabilistic artifact, and meaningless in and of itself. (keep in mind that a small percentage of randomly filled out answer sheets will have perfect scores!)</li>
<li>colleges recognize that there is no statistical difference between a score of 2400 and a score of 2300 is meaningless? (the SD of any individual test is greater than 100!)</li>
</ul>

FWIW, we already know what these other characteristics are that colleges look for. How could someone not know?

cghen · August 7, 2007, 4:20pm

tokenadult, you’re mistaking rarity for quality. While those are often correlated, they are hardly the same.

It’s (sort of) analogous to arguing that people who perform an unassisted triple play (<a href=“http://en.wikipedia.org/wiki/Unassisted_triple_play[/url]”>http://en.wikipedia.org/wiki/Unassisted_triple_play</a>) in baseball are better players than those who pitch perfect games (<a href=“http://en.wikipedia.org/wiki/Perfect_game[/url]”>http://en.wikipedia.org/wiki/Perfect_game</a>), because the former is rarer than the latter. In reality, the former is rare because of reasons external to the quality of the player, which is quite similar to a student who scores a 2400 on the SAT versus someone who scored a 2300 with other accomplishments.

What I mean is that it’s rare for a student to get a 2400, but it is highly dubious to suggest that a 2400 scorer is any more qualified (a priori) than a 2250-2300 scorer, since a few ‘random’ mistakes can lead to the latter and arbitrary preparation to the former. Colleges understand this, and even the most meritocratic of schools (like Caltech) make no distinction of scores near the high end of the spectrum.

Indeed, I would highly question the admissions policies of any school who used the SAT more than just a tool to weed out obviously unqualified applicants.

mammall · August 7, 2007, 4:37pm

What is the real agenda of the author of this thread? It feels as if some people are unhealthily fixated on 2400. Yes, a perfect score is rare. Yes, there are a few extraordinarily gifted kids who can do it in a single sitting. I just get tired of the long tracts trying to deny the extraordinary nature of the accomplishment. Also, I don’t buy the “random” argument. A perfect score is a perfect score representing about four intense hours of maintaining amazing poise under severe stress. Of course, such students deserve to be first in line to the best colleges, provided their grades and disciplinary records are good. The real question is whether insitutions like Harvard that seem to enjoy bragging about rejecting pefect SAT scorers will endure much longer as the accepted academic leaders. If they aren’t getting the best minds (and yes I do believe the SAT demonstrates intelligence!) then eventually their graduates will no longer be regarded as particularly special. College admissions has degenerated into nonsense in this country.

epiphany · August 7, 2007, 4:49pm

^^ Then surely you’ll want to go to a different country, where college admissions is supposedly not nonsensical.

And btw, there are accomplishments which are far more extraordinary than 2400 SAT scores, and such extraordinarily accomplished students are recognized as such by top U’s & admitted to them every year. In fact, those U’s only reject 2400’s when other categories of “extraordinary accomplishments” are not present, and/or when the application for any reason does not demonstrate parallel & differentiated kinds of achievement, talent, INTELLGENCE, and an individualized, personalized reason to attend that U. That is even true at publics like Berkeley, which can & have (recently) rejected 2400’s for the simple reason of an arrogant essay, unconvincing statement of purpose, etc.

kluge · August 7, 2007, 5:05pm

Tokenadult, I think the previous posts on the lack of any significant distinction between a student who scores 2400 and one who scores 2300 - both scores within the top 1/2 of 1% of all student test-takers - pretty much explain the situation. If you just take that 1/2 of 1% - 2300+ SAT, 34+ ACT - you’ve got a pool of about 12,000 students - or twice the combined freshman classes of the schools you listed. While some of those students will elect to go to Berkeley, or Williams, or Michigan, or Olin, or San Diego State (as one of my children swears is the case) there’s still a lot of them who are going to apply to the short list of school you identified. So naturally they won’t take 'em all. (I know one in that bunch who pretty certainly won’t be a freshman at any of those schools next year.) Each of those schools will reject some 2300’s, and they will also reject some 2400’s. Most applicants in that cohort will catch on somewhere, though.

cghen · August 7, 2007, 5:09pm

mammall,

Even if you feel that the US universities should be more numbers oriented (which many people, including myself would adamantly argue against), you must at least agree that the SAT is not designed to fit this function. Simply put, the SAT is not nearly challenging enough to significantly distinguish between top scorers. When the difference between a 2400 and a 2380 is, at most, one trivial algebraic mistake, one misreading of a nuance in a passage, or one mistaken bubbling of an answer, it seems pretty clear that the SAT cannot adequately distinguish between the 99th, 99.9th, and 99.99th percentiles of students, which is wholly needed to base admissions off of exams. What tends to work better in this regard are tests associated with the various math and science olympiads, which many schools value quite highly (though, obviously not exclusively).

tokenadult · August 7, 2007, 5:30pm

Post #115 asks,

My agenda is to get high school students to focus on other forms of preparation for an application to a competitive college besides just gaining higher test scores. I’ve been spending a lot of time on the test-specific forums here on College Confidential recently, and I get the dismaying impression that some students think that clearing some particular threshold of scoring is going to be their guarantee of getting into their college of choice. I hope every student applying to college this year, next year, and in subsequent years is aware that a lot of other issues matter in college admission in the United States. But I fear that some students, particularly those whose parents received their college educations in other countries, may overemphasize test prep to the detriment of other forms of college preparation. I have been HORRIFIED, on the summer programs forum here on CC, to see students say, “My mom won’t let me apply to [famous, highly educational] summer program because she wants me to use the summer to practice for the SAT.” That’s a misplaced priority, in my humble opinion.

The posts immediately following my post #112 have some good comments about why a score of 2400 as such (or 36 as such on the ACT) is not particularly decisive in a college application. I agree that a student is wasting effort to try to turn a (for example) 2360 into a 2400, but some students seem to want to do such things.

I’ll try to sum up some recommendations to students in another reply to this thread in a while. Meanwhile, keep it coming.

marite · August 7, 2007, 5:48pm

Given your agenda, I would avoid characterizing colleges’ admissions policies as “having to dig deeper” in order to fill their entering classes. As so many others have pointed out, there is no difference between a 2300 and 2400. No digging deeper is necessary to prefer a 2300 who exhibits other strengths over a 2400 with no other strength. In the example provided by Quiltguru, a higher scorer was passed over in favor of someone whose lower score was more than compensated for by greater achievements in academic ECs.
This is something you might want to share with your young acquaintances.

PS: One of S’s good friends, a budding physicist, is from your state.

How do top scorers on tests fail to gain admission to top schools?

CONNECT WITH US