The Misguided War on the SAT

[mtmind]

Not by @MITChris

[sbinaz]

@Chekov

Ok. Fine. Can you explain it to me like I am a six-year old why, in your opinion, MIT decided to go back to requiring the tests?

the Your College Bound Kid podcast interviewed an MIT admissions person in 2022 on this topic. You could hear it straight from the horse’s mouth!

Interview 90: Chris Peterson and Kellen Manning on "Understanding MIT" - Your...

[DadOfJerseyGirl]

@sbinaz

You could hear it straight from the horse’s mouth!

@MITChris (one of the MIT admission officers you mentioned) has very helpfully provided his views in the posts above.

[anaray]

TO students have not graduated yet though.

[3SailAway]

I think it’s clear that Mr. Leonhardt believes tha SAT to be a predictor of the achievements listed:

These students, in turn, can produce cutting-edge scientific research that will cure diseases and accelerate the world’s transition to clean energy. The students can found nonprofit groups and companies that benefit all of society.

Administrators at elite colleges have justified their decision to stop requiring test scores by claiming that the tests do not help them identify such promising students — a claim that is inconsistent with the evidence.

Where is the evidence that a person who scores high on the SAT, or gets good grades in college is more likely to change the world for the better in the above ways?

The data in the article shows that high scorers are more likely to be “Attending a prestigious graduate school” or “Working at a prestigious firm.” I don’t think people who are “Working at a prestigious firm” are more likely to start nonprofits than those whose goals do not include being employed by a prestigious firm.

@Chekov

Even so, academically excellent students are indeed more likely to succeed in academic endeavors, and so if one concedes that the answer to question 1 is in fact “academic excellence” then perhaps we should not discard the tools that we have to achieve them because social justice advocates find them “not politically correct”.

I hope higher education has many goals. Academic excellence is a worthy goal, but so much more than success in standardized tests/grades goes into being (for example) a groundbreaking immunologist or globally influential alternative energy entrepreneur.

An article in today’s NYT is about a successful and growing village for the homeless in Austin, Texas. It was built and managed by the nonprofit Mobile Loaves & Fishes, founded by Alan Graham who dropped out of college: https://www.nytimes.com/2024/01/08/headway/homelessness-tiny-home-austin.html?searchResultPosition=1

[Chekov]

@3SailAway

Academic excellence is a worthy goal, but so much more than success in standardized tests/grades goes into being (for example) a groundbreaking immunologist

I do not believe anyone is saying that standardized tests are the be-all and end-all of… well… anything.

But plenty of people eschew any objective measures of academic merit on ideological grounds despite evidence of their usefulness.

Of course there’s more to becoming a great scientist than being able to read. But learning to read it a good start.

@3SailAway

founded by Alan Graham who dropped out of college

So… not everyone has to go to college?

[Data10]

@3SailAway

The data in the article shows that high scorers are more likely to be “Attending a prestigious graduate school” or “Working at a prestigious firm.” I don’t think people who are “Working at a prestigious firm” are more likely to start nonprofits than those whose goals do not include being employed by a prestigious firm.

This is largely a misinterpretation of the Chetty study. The Chetty study defines working at a prestigious firm as working at firm that employs a disproportionately large number of Ivy+ grads. SAT score in isolation explained <2% of variance in employment at Ivy+ heavy firms compared to 0.2% for GPA. Neither stat explains much about which students work at the firms, nor does the combination of SAT + GPA. Instead what had by far the most impact was high school fixed effects. Specific numbers are below:

Chance of Ivy+ Grad Working at Firm That Employs a Disproportionate Portion of Ivy+ Grads
HS GPA – 0% of variance explained
SAT – <2% of variance explained
SAT + GPA – <2% of variance explained
Above + Legacy + Race + Gender + Athlete + Parents Income-- 2% of variance explained
Above + cross variables – 5% of variance explained
Above + high school fixed effects – 31% of variance explained

Chance of Ivy+ Grad Attending an Ivy+ (or 1 of 4 publics) for Grad School
HS GPA – 0% of variance explained
SAT – 1% of variance explained
SAT + GPA – 1% of variance explained
Above + Legacy + Race + Gender + Athlete + Parents Income-- 3% of variance explained
Above + cross variables – 5% of variance explained
Above + high school fixed effects – 27% of variance explained

Looking at the stats above, I don’t think the big takeaway is that SAT’s 1-2% variance explained is a more powerful predictor than GPA’s <1% of variance explained. I think the much more notable finding is the strength of high school fixed effects. High school fixed effects are a far stronger predictor than anything else that is analyzed, including both SAT and GPA.

The author implies high school fixed effects can be interpreted as the difference in outcomes between students who have the same GPA/SAT stats, same demographics, and same parents income; but attend different high schools. Some high schools have a far higher rate of kids working at Ivy+ firms or attending Ivy+ grad school than others.

The analysis implies that if a college wanted to admit kids with prestigious firm or grad school aspirations, then they might favor students from particular high schools. Perhaps the types of students who choose prestigious high schools are also the type who are likely to seek prestigious outcomes post college. Or they might look more in to why those high schools have such high rates of students with with prestigious aspirations and figure out what non-stat parts of the application are likely to reflect that such as combination of students major + future plans, essays, interview, and life background. With SAT score explaining <2% of variance, it does not suggest test optional vs test required would have a large impact from the score itself, but might indirectly have effects by admitting more kids from different HSs or with different future aspirations.

[sbinaz]

One thing that I don’t quite follow is why is it such a big deal that some schools require test scores and others do not? Why is it a problem that a school like MIL wants to require SAT/ACT but others schools do not follow suit?

Each college/university has its own institutional goals. MIT has a system that works for them. Caltech has a different system that works really well for them. Both institutions produce stellar scientists and engineers.

There’s more than 1 way to skin a cat.

[BUMD]

@anaray

TO students have not graduated yet though.

I don’t think TO students have anything to fear about not graduating.
The minimum grade to graduate is a C- or a 1.67.

There is a lot of cushion.

[Chekov]

Caltech only went TO in 2020. Few TO grads have been produced so far (only those who graduated early).

Signed,
Capt. Obvious

ETA: My mistake. Of course, Caltech went test-blind, not TO.

[3SailAway]

@Chekov

I do not believe anyone is saying that standardized tests are the be-all and end-all of… well… anything.

But plenty of people eschew any objective measures of academic merit on ideological grounds despite evidence of their usefulness.

I do think that Mr. Leonhardt is suggesting in his article that people who excel at taking standardized tests at age 17 are more likely to benefit society.

As far as eschewing any objective measures, I don’t think colleges should be forced to be test blind, but that’s not happening outside of California, right?

[BrownRiceS24D27D29]

@sbinaz

MIL wants to require SAT/ACT

My kids’ suitors will need to tell me their AMC and AIME scores.

[Chekov]

@3SailAway

I do think that Mr. Leonhardt is suggesting in his article that people who excel at taking standardized tests at age 17 are more likely to benefit society.

I don’t believe there is much disagreement on that particular point. In fact, you could measure it a whole decade earlier.

“The American Psychological Association’s report Intelligence: Knowns and Unknowns states that wherever it has been studied, children with high scores on tests of intelligence tend to learn more of what is taught in school than their lower-scoring peers.
…
While IQ is more strongly correlated with reasoning and less so with motor function, IQ-test scores predict performance ratings in all occupations.”

Lots of other interesting correlations:

Intelligence quotient | Social correlations

@3SailAway

As far as eschewing any objective measures, I don’t think colleges should be forced to be test blind, but that’s not happening outside of California, right?

I am sure some will say that not much of anything is happening outside of California

[Chekov]

@BrownRiceS24D27D29

@sbinaz

MIL wants to require SAT/ACT

My kids’ suitors will need to tell me their AMC and AIME scores.

Now, that is not a bad idea!

[mtmind]

@Chekov

@3SailAway

I do think that Mr. Leonhardt is suggesting in his article that people who excel at taking standardized tests at age 17 are more likely to benefit society.

I don’t believe there is much disagreement on that particular point. In fact, you could measure it a whole decade earlier.

Great idea. Let’s test kids at the age of 7 to determine where they will attend college. Or if they get to go to college. What could go wrong?

[Chekov]

Whose idea is that?

[DadOfJerseyGirl]

@mtmind

I have been led to believe that if M.I.T. says it is so, it must be so for every school and every situation.

@mtmind

Great idea. Let’s test kids at the age of 7 to determine where they will attend college. Or if they get to go to college. What could go wrong?

Who said either of these things? Are we reading the same thread?

[cinnamon1212]

I believe @mtmind is being sarcastic to make a point.

[Mwfan1921]

@Chekov

Caltech only went TO in 2020.

CalTech is test blind not TO. Still no test blind grads yet tho, but I doubt CalTech will back off test blind admissions. Time will tell.

[MITChris]

I think everyone college/university should do what is right for them, and personally, I avoid weighing in on making prescriptions here. But our position (I speak collectively and personally) has been that test-optional is sort of the maximally inequitable. As Stu said in this piece: What Does an SAT Score Mean in a Test-Optional World?

Amid the confusion, high-school counselors are struggling to find new signposts to guide their students. One problem, counselors told me, is that the information they get from colleges isn’t standardized. Some report the percentage of applicants who applied without test scores but not the percentage accepted; others report the inverse. And while colleges report the SAT and ACT score ranges of enrolled students, they don’t indicate which percentage of the incoming class submitted those scores, leaving students to wonder how much the figures have been inflated by those with high scores who bothered to submit them. Two years in, counselors have no idea: What is a good score? Do I submit a score or not? And if so, should all colleges on my list get my score? Schmill tells me he gets those same questions from friends whose children are applying to other colleges. “I never had a good answer,” he said. “Like, I have no idea.”

If test-blind works for Caltech, then more power to them, especially for not hedging by going TO.