National Merit Cutoff Predictions Class of 2017

Paying for College Financial Aid (Need-Based, Merit Aid, Loans)
1981

@DoyleB - I think (unfortunately) we’re pretty much in agreement.

Are we misinterpreting the data? Did CB make one (or more) mistakes?

[Note - I ramble on quite a bit. If anyone wants to skim, I actually think my final paragraph might be somewhat interesting…]

The fact that the concordance table and the SI table don’t agree leads me to believe that those tables were generated with different techniques. (If they just generated the concordance table directly from the percentile table, then obviously they would agree.)

Even though they define their concordance table to essentially match up percentiles, a logical way to try to generate it (especially for a new test) is to have their research group take both the new PSAT and an old one (or whatever they’re trying to concord it to).

I guess one explanation would be the following. Say people in their research group who got a 214 on the new psat tended to also get a 214 on the old (I’m assuming that’s what the concordance table says) - but only 0.5% got >= 214 on the new (while 1% got >= 214 on the old when it was given for real). They then have a discrepancy. The concordance table would say that 214 → 214 - but the percentile table would say that 214 was 99.5% (instead of 99%). Maybe the research group just didn’t have the correct fraction of smart(est) kids? Would that just imply that the percentiles are wrong, and concordance is right? Not sure why they wouldn’t adjust for that. Maybe seems too simple minded to me.

Another high-level possibility. The research group prepared for the old PSAT, but not the new one. This means that current students (who presumably prepared for it) would find the test easier than the research group. But this should affect both concordance and percentiles - not sure why that would make concordance accurate but percentiles wrong.

Also - maybe they wanted the scores to be directly comparable to old PSAT scores. I actually think this is a stated goal of theirs (at least PSAT should match what you would get on the SAT). Maybe they started with that goal, and so set up the raw score translation table such that final scores would somewhat match previous PSAT scores. And then maybe they used some other technique to try to estimate the percentiles (and just failed)?

Obviously the anecdotes make us think that the concordance tables are correct (and the percentiles are not). The biggest problem that us percentilers :slight_smile: have is that getting accurate percentiles seems SO EASY!!! Some people say that they know the accurate percentiles, and just don’t want to release them, because DRAMA. I don’t buy that. Could it be possible that the scores are distributed across data centers, and so they actually can’t put them all together and sort them? That seems mind-bogglingly stoooopid to me - but I work at Google. Maybe they’re a bunch of statisticians and educators - and their IT stuff isn’t up to snuff.

I guess a final possibility (I should stop writing already) is that they do know the accurate percentiles for each year, but prefer to compare to the previous year, because they feel it’s more accurate / helpful somehow. If the population of test takers changes dramatically from year to year - maybe it’s more helpful to say how you did compared to the population last year?

OK, one last thought - I promise - that could explain both the concordance tables and the percentile tables being correct (though not the anecdotes). Maybe the kids taking the test this year are actually not as smart (no offense :slight_smile: ) as either predicted and/or last year. I mean, if I understand things properly, I think that’s exactly what the concordance table and percentile table are saying. That the top 1% of kids last year would get a 214 on this test (and also the old test) - but only the top 0.5% of kids this year got a 214… (Put another way, maybe CB predicted that from studying specifically for the new PSAT, scores would go up a lot, and maybe they overcompensated.)

1982

@thshadow your last paragraph WAS interesting. I’ve always kinda thought the anecdotes may be outliers doing what outliers do…draw attention to themselves…and they don’t really represent a nationwide phenomenon.

1983

In Texas, our school has typically had about a dozen NMSF. This year we have 30 juniors at 99% or greater

1984

What if anything can we learn from this thread: http://talk.collegeconfidential.com/sat-act-tests-test-preparation/1849778-commended-psat-score-for-class-of-2017.html#latest
Perhaps updates the table with actual scores that was made a few days ago?

1985

@thshadow, " Some people say that they know the accurate percentiles, and just don’t want to release them, because DRAMA. I don’t buy that." I agree. I think CB is pretty busy with lots of redesigns, concording, online technology. I think they are not worried about people predicting state cutoffs correctly. Lots of other things to occupy their time.

But what I really what to touch on is the subject of Field Tests. Ok, so my daughter’s high school was selected to do the field test on the new PSAT. Many people may not realize this, but the Reading passages between the field test and the real test given in October 2015 were EXACTLY the same. That hard Douglas passage people have complained about - was on the field test. All those students had a slight advantage over the general test takers on the October, 2015 PSAT because they had already read those passages a few months earlier. Now, of course, the questions were different between the tests, but passages were the same. My daughter, in her infinite wisdom, elected to sleep in that Saturday morning and not take the field test with many of her classmates - that came back to bite her. She didn’t get the advantage her classmates got.

Now, the field test students would have been sophomores. They took the old psat in October of 2014. However, many of them did not really prep and study hard at that point for PSAT. They were only sophomores and their scores didn’t count for anything. Then the sophomores took the new psat as part of the field test. I don’t remember what month that would have been. Maybe Jan or Feb of 2015 - but can’t remember really. But likely a few months between old psat and new psat as part of the field test. None of the students would have studied for the new psat/field test. No practice tests were available. So basically the sophomores didn’t study for the old psat or the new psat/field test.

I’m thinking the field test students are the students cited as being the 90,000 students in the “research study”. I would expect nearly all the students to have done much better when they took the official PSAT in October, 2015. They were juniors, “smarter” after another year of school and many would have done prep for the real test. Plus that advantage of having already read the Reading passages once before.

Maybe the field test students doing much better on the October, 2015 PSAT vs the field test psat isn’t/wasn’t fully accounted for by CB - or was over compensated by CB.

1986

@mphill1tx, can you be more specific, 30 at 99% of National, User, TS, SI?

There really are 4 different percentiles people can specify.

Your general “99%” reference maybe the only way scores were quantified to you, but more specificity helps.

1987

@mphill1tx – the percentile info being reported makes it hard to predictt as the terminology changed - would help to get some SIs if possible.

1988

@Speedy2019 wrote: " I would expect nearly all the students to have done much better when they took the official PSAT in October, 2015. They were juniors, “smarter” after another year of school and many would have done prep for the real test. "

Which brings us to another part of the compass article - the part where the “official, accurate, they know what they’re doing” college board tables show sophomores performing almost as well as, and in some cases better than, the juniors who took the test. I found that to be pretty dubious as well.

1989

@micgeaux - that’s a fascinating idea.

I just looked at the data briefly, and at first glance, it seems to be on the up-and-up. IOW, if you assume the top 100 is the top 100 kids (sorted by 1 single metric) instead of sneakily taking the top 100 math coupled with the top 100 RW, the numbers seem reasonable. If they had “cheated”, the average of the lowest kids would be unreasonably low. (Because the lowest “kids” are actually amalgams of scores.)

For example, at Wheeler, the worst 45 kids would have averaged 1078. At Sprayberry, the worst 6 kids would have to average 738. At Hillgrove, the lowest 107 kids would have averaged 972. At Campbell, the lowest 23 would average 838. At Walton, the “bottom” 501 (out of 601) averaged 1158 (vs. 1207 for the whole school).

I don’t really know how to estimate what’s “reasonable” (you’d have to guess both the std dev for the bell curve as well as the correlation between math and reading scores), but the above don’t seem too crazy to me. I would say that the chance that they didn’t cheat (or even that they did cheat but the scores are so correlated that it doesn’t make a big difference) is quite high.

1990

@CA1543 - I think we can learn that CB just needs to release the actual percentiles for the actual test-takers on the Oct. 2015 exam. I think that all of this speculating and number crunching is messing with too many heads! @-)

1991

@mphill1tx Texas has roughly a 3-1 ratio of commended to nmsf, so 30 in 99% sounds about right for a school with 12-15 nmsf.

1992

@thshadow wrote:

“I guess one explanation would be the following. Say people in their research group who got a 214 on the new psat tended to also get a 214 on the old (I’m assuming that’s what the concordance table says) - but only 0.5% got >= 214 on the new (while 1% got >= 214 on the old when it was given for real). They then have a discrepancy. The concordance table would say that 214 -> 214 - but the percentile table would say that 214 was 99.5% (instead of 99%). Maybe the research group just didn’t have the correct fraction of smart(est) kids? Would that just imply that the percentiles are wrong, and concordance is right? Not sure why they wouldn’t adjust for that. Maybe seems too simple minded to me.”

The definition of concordance is to get the percentiles to match up. If a score on one test is at 50.00%, it concords to whatever score on the other test was at the 50.00%. If a score on one test is at the 99.50%, it concords to whatever score is at the 99.50% on the other test. That’s it - the actual value (number) of the score never comes into play.

1993

@thshadow - let’s say that your last paragraph of #1980 is correct. Does that mean that the page 11 SI table represents actual results as opposed to “research study” results? Because the CB person on Twitter responded “research study” I thought. Not that this was necessarily the factually correct answer.

1994

@mphill1tx, your school is one of the below?

13 - Jesuit Catholic (boys - private)
13 - Plano Senior (Plano ISD)
12 - Allen (Allen ISD)
12 - Plano East (Plano ISD)

From http://dallas-area-schools.blogspot.com/2015/09/2016-national-merit-semifinalists.html

1995

That was quite interesting. Also, 90,000 kids is like 5% or 6% of kids who took the test! That’s a pretty big sample.

Is it possible that some of the “anecdote” schools were also field test schools?

1996

@mphill1tx “In Texas, our school has typically had about a dozen NMSF. This year we have 30 juniors at 99% or greater”

Is there anyway you can get scores from those people? are they looking at their national percentile? If so, we don’t want to go by that…SI scores would be helpful.

1997

Been skimming this thread. Haven’t pored over the data nearly as much as many of you and not convinced anyone can figure out what is wrong from the released info.

@Speedy2019 Wow, you are saying they gave the field test to 80,000 sophomores who then got to do the exact same readings on the real test? I don’t see any justification for this. I had assumed they would field test with juniors who could no longer gain an unfair advantage.

1998

Whoa there, @Speedy2019! This is very interesting! The fact that they administered the same paragraphs (but different questions) between the field test and real PSAT test. However, if the test was administered to sophomores, they couldn’t comprise the research sample, as that was intended to be among 11th graders only. But if the field test was given to juniors, I would think it’s unethical of CB to administer same paragraphs in ERW sections. That way the juniors in the research sample will have seen the difficult english context earlier, giving them an unfair advantage. Either way, we don’t really know what the research sample is unless CB clarifies it further. Perhaps it’s those students whose scores were disqualified when caught cheating on SAT?!!

1999

@thshadow, from memory, CB said 90,000 9-12 graders. Presumably, the research study also accounted for PSAT 8/9 and PSAT 10. Not all of the 90,000 was for PSAT/NM. My understanding.

2000

Wow, I’m really surprised that they would reuse the same reading passages from the field test on the real PSAT. I had expected that they used the test that they later released as the one practice test. I guess these were the passages on the Oct. 14 test? Or perhaps some mix of Oct. 14 and 28?