National Merit Cutoff Predictions Class of 2017

[F1RSTrodeo]

My take on the concordance tables. I seem to compress at SI 217/218 and not as much as Testmasters

                   2014 My 2015

Alabama 209 211
Alaska 206 209
Arizona 215 216
Arkansas 204 207
California 223 222
Colorado 215 216
Connecticut 220 219
Delaware 216 216
DC 225 223
Florida 214 215
Georgia 218 218
Hawaii 214 215
Idaho 208 210
Illinois 215 216
Indiana 213 215
Iowa 208 210
Kansas 213 215
Kentucky 210 212
Louisiana 211 213
Maine 211 213
Maryland 222 221
Massachusetts 223 222
Michigan 210 212
Minnesota 214 215
Mississippi 209 211
Missouri 209 211
Montana 204 207
Nebraska 209 211
Nevada 211 213
New Hampshire 213 215
New Jersey 225 223
New Mexico 208 210
New York 219 218
North Carolina 215 216
North Dakota 202 206
Ohio 215 216
Oklahoma 208 210
Oregon 215 216
Pennsylvania 217 217
Rhode Island 212 213
South Carolina 211 213
South Dakota 202 206
Tennessee 212 213
Texas 220 219
Utah 206 209
Vermont 214 215
Virginia 222 221
Washington 219 218
West Virginia 202 206
Wisconsin 208 210
Wyoming 202 206

[CA1543]

From Post 1897:

Can we think about where the state cutoffs might fall here -based on this chart by @likestowrite (thank you) & the 2014 understanding scores report (link
http://www.bernardsboe.com/UserFiles/Servers/Server_3096886/File/Jill Shadis/Ridge Counseling/Standardized Testing/Understanding 2014 PSAT-NMSQT Scores.pdf

2015 SI – % – %ile moved up 1% to account for definition change / 2015 predicted/estimated state cut offs based on prior percentiles of state cut offs

228 – 99±- 99+
227-- 99+ 99+
226-- 99+ 99+
225-- 99+ 99+
224-- 99+ 99+
223-- 99+ 99+ DC?
222-- 99+ 99+ DC?
221-- 99+ 99+ Cal? DC?
220-- 99+ 99+ Cal?
219-- 99+ 99+ Cal?
218-- 99+ 99+ NY/ TX / GA / Wash?
217-- 99+ 99 – NY/ TX / GA / Wash?
216-- 99+ 99 - NY/ TX / GA / Wash?
215-- 99+ 99
214-- 99+ 99
213-- 99 98
212-- 99 98
211-- 99 98
210-- 99 98
209-- 99 98
208-- 99 98
207-- 99 98
206-- 99 98
205-- 99 98
204-- 98 97
203 – 98 97
202 – 98 97
201 – 97 96
200 – 97 96
199 – 96
198 – 96

Thoughts???

[Mamelot]

@Shelt29 I’m way behind in reading posts at this point but can respond in part to your #1912:

So right away let’s eliminate any comment or concern that mentions “National Representative”. It simply isn’t a population of students that pertains to NM questions. Right there - that takes care of some of the excess referred to in the Compass Report. Sure they have issues with the “National User” but they aren’t, apparently, inflationary concerns - they border more on concerns about the research study and perhaps lack of accountability as these are buried deeper in the online report. So we need to focus on User Percentiles, and also the page 11 SI Table (with as-yet-not-defined-precisely percentiles). And we need to focus on some of the so-called “inflation” that results from changing the definition this year to “greater-than-or-equal-to” from the previous “greater-than” that applied to percentiles. And, of course, the concordance tables since these, while preliminary, are still the best method we have of understanding how 2015 scores relate to historical cut-off’s.

Theoretically, all of these should mesh. The User percentiles should look reasonable when compared to the SI percentiles (small differences due to Total Score vs. SI, of course) and all should “concord” consistently to previous scores that mesh with previous percentile tables. The concordance tables, after all, are supposed to concord on a “same percentile” basis. And one important assumption which you touched on: All of these percentile tables should assume a stable population of test takers (not 1.5 million one year and 3.0 million the next, etc.).

The challenge has been to try to see if all the tables DO mesh. I’m finding that sometimes they do, and sometimes they don’t. It’s an important question because to me it’s very hard to make a prediction if we can’t trust or understand the tables.

@likestowrite has done some interesting work showing how previous and current SI percentile tables may, indeed, mesh if we account for the change in percentile definition (from “greater-than” to “greater-than-or-equal-to”). See post #1897 and @CA1523’s post 1921 above. Personally, I like this approach - it’s simple and doesn’t tie us up all in knots about whether CB is being honest, or trying to “inflate percentiles” or tricking everyone with the wrong student population, or whatever. It simply addresses the question of how to reconcile previous tables with the current table and provides an answer. So far so good.

When trying to concord scores, that’s where there can be a problem, My earlier mistake is that I concorded last year’s 202 to a 202 this year - oops!, it’s more like 203 - 210. Obviously some work needs to be done in the area of concordance. Concordance worked for my D3’s scores but not in lower areas of the curves (the areas that everyone is saying shows an inconsistency, which should be no surprise!).

So - Bottom Line: preliminary conclusions about the commended # is that a 202 - 207 (or so) makes sense at this point. I think it’s closer to 202 than to 207 but the concordance tables don’t support that quite (not sure yet if I’m not working correctly with them or they are just not very accurate). I wish I could be more exact but this is estimation which is anything but an exact science. :-?

[CA1543]

@F1RSTrodeo – can you share your methodology for your predictions please?

@123field – Thanks so much for all of that data from prior years!!Very useful to try to estimate what might happen for various states.

[LivinProof]

@CA1543 I believe @DoyleB showed that the CA cutoff was always (all 4 years he looked at) at the highest 99% bracket, therefore the 217 score on this revised table.

[Pickmen]

@Shelt29 1912 I think I get what you are saying, and at the risk of stating the obvious, I think it helps to understand we are dealing w/ 3 different groups of percentiles:

(1) the reported SI, User, National and TS each tester received and CB published (p11)
(2) the concorded percentiles from the prelim charts and prior PSATs
(3) the de facto or FIXED percentile created by a defined population group of 50k testers out of 1.5/1.7 mill who will receive SF or commended status depending on their state cutoffs (about 3% or top 97% of the test takers).

Trying to make sense or reconcile these 3 groups has proved impossible UNLESS one accepts that the SI percentiles each test taker has received is based on a national or 3.5 mill reference population not 1.7. mill. Then 3% in the top 99% makes sense because this would cover 35k kids not 17k. The remaining 15k would be in the top 98% and so the concorded tables now make sense and Testmasters cut off for commended at 210 also looks reasonable.

This still leaves us with the problem of trying to reconcile the user percentiles that CB reported in their guide and on testers’ scores as they do not concord and I am not sure how they will explain their way out of that. Also they imply their SI applies to 1.5 “current” test takers so providing an SI that is based on 3.5 mill is misleading and I do not know why they would do that. But it’s the only thing making sense to me right. I wish it weren’t so or some one could tell me how wrong I am.

[LivinProof]

@CA1543 It was post #1526 where the CA cutoff history was shown.

[F1RSTrodeo]

Used data from @thshadow and an idea from another poster. Compared mode, median, mean, min and max for each combination of SI#. Mode, median and mean correlated rather nicely.

[micgeaux]

Based on @123field, the 99+ has 14-17 spots; the 99s have 9-10 scores. I would decrease the number of 98 percentile scores and increase the number of 99 percentile scores. Totally unscientific, but I think you have too many 98s.

[CA1543]

@LivinProof – As someone has pointed out there is compression at the top because the scale range is smaller so if more students have to fit into fewer SI’s, it may also tend to push up a state’s SI cut off because of the limited allotment each state gets. I am wondering what others think about that possibility. You can see the ranges that used to exist for the percentiles about in post 1919 by @123field . Test Masters predictions are looking more realistic and based upon an approach we can basically replicate it seems. http://collegeadmissions.testmasters.com/update-psat-scores-cut-national-merit-2016/

Of course there is no way to be precise at this point & likely never until the cut offs are issued in September. But if we can agree on a concordance of SI’s we think is reasonable then as data & state reports come it, we can think about the effect of new info.

[F1RSTrodeo]

@CA1543 - Looks like my predictions are matching up with the top prediction of the table you produced at post #1921

[Speedy2019]

@Pickmen, I don’t see anything from testmasters indicating they assumed 3.5M to arrive at a 210 commended. My understanding is they took a “conservative approach” when doing their concording. The concordance tables result in a range based on how the 3 sections are scored, right? I think they picked the highest number from that range. As @Mamelot noted, a 202 concorded to a range of 203-210. Testmasters went with 210.

If they took a liberal approach instead of a conservative approach, (ie, 203 commended) their cutoff predictions may match better with what the SI % table is showing.

Disclaimer, I have examined all the testmasters prediction to actually see how the cutoffs would look with the liberal approach, but for the commended number the liberal approach is close to the SI% table.

[Pickmen]

@Speedy2019 1931. Correct. Testmasters got 210 cutoff based on concordance. My point rather was about reconciling the SI percentiles (p11) vs the fixed percentiles created by 50,000k testers as well as the concordance tables. All 3 only make sense if we accept that an SI of 209/210 is 99% if the population of test takers is 3.5 mill for 2015 that concords to 97% for a population of 1.5 mill test takers for 2014 and prior. Sorry if this is still not clear. As I said, I hope I am missing something here.

[F1RSTrodeo]

I agree with @Speedy2019 … Testmasters put out a conservative estimate for commended. 206 seems to be the sweetspot.

[Speedy2019]

@Pickmen, would “all 3” make sense better if a liberal approach was used instead of a conservation approach - meaning 203 for commended instead of 210 (based on a concorded 203-210 range)?

Or closer to the mid-point of the range, such as 206 that @F1RSTrodeo suggests?

[DoyleB]

Very long post, but here goes. Bear with me.

There’s been a number of posters here relying on the change in percentile definition to explain fairly large changes that they are making when developing their own versions of the SI percentile table. That is incorrect. I’m going to state this one more time - all the change in percentile definition does is shift the entire table by one row. In the middle of the table (the part of the distribution we don’t care about) this can change the reported percentile by 3 or 4 points. But at the top it changes it by very small amounts. The change in definition doesn’t change all the 99s to 98s for example - that’s not how it works. And it doesn’t change the percentiles by specific values, like 1% or 2%. It changes by one row.

Exercise 1: You have last year’s SI table, which used definition A. You want to change it so that it uses definition B instead. It’s in an excel spreadsheet, and there are two columns. The first column is SI, the second column is percentile. Here’s how to do it:

Leave the SI column alone. Grab the entire column that shows the percentages (99+, 99, 98, …, 3, 2, 1). Shift the whole column down by ONE row. This shift down leaves one empty spot at the very top of the percentile column. Fill that empty spot with a 100.0%.

Ta da! You now have last year’s table, but it uses the new definition of percentile instead of the old definition.

Exercise 2: You have this year’s table, and want convert it so it uses the same definition as last year. Just shift the entire percentile column up by one row. That’s it - you’re done. (Well, almost done. You have one extra entry at the top of the column. Just delete that. And fill the empty spot at the bottom with a 0%).

Notice that this shift didn’t make a dramatic difference. If, to create your new SI table, you’re shifting by more than one row, you need to develop another explanation for the large shift. Something else entirely must be going on.

It appears to me that this year’s SI table needs to be shifted by a LOT to be reasonable - it shows 205 as being the first of the 99s, and anecdotal data shows that the first 99 is likely much closer to 213. Changing the definition of percentile only shifts a 204 to a 205. For the table to be that far off, either it is referring to a significantly different population than the 1.7 million test takers, or that CB royally screwed up the “research group”.

I’ve got more to say about “national” vs “user”, and CB’s estimates of where the phantom “non test takers” would fall into the distribution, but I’m already falling into the “too long, didn’t read” category. For now I’ll just say I have no idea how the SI table got so messed up.

Still waiting for the mystery to be solved.

[Speedy2019]

oh, the disclaimer should have said “I have NOT examined all …”

[Pickmen]

@Speedy2019 – 1934. I don’t think you can pin a # down. I was just trying to see how all the 3 percentile groups can make sense, which they do IF you accept in 2015 the SI refers to 3.5 mill testers and not 1.7 mill. That said possibly a 206 could work if it comes within the high 98% range of 3.5 testers. I just don’t know. I’m speculating not crunching numbers. Also, the big PROBLEM with this speculation is that CB published user percentiles that are supposed to reflect a 1.5 mill population even if derived from sample group. They make no sense with concordance and blow a big hole in my attempt at reconciling the conflicting data. FWIW I think 210 is too conservative. Concordance works basically with 209 SI too and I think other posters have shown it works up to a point at the 202/203 level.

[dallaspiano]

@CA1543, very good way to data in term of SI percentile ranking
@123field, excellent records
But and a lot of but

1. If believe data behave like Testmaster’s estimate, then use Testmaster’s results
   2- Assumptions of end range of 99+% (corresponding to SI 223), that mean historical and traditional +%range we used to have at 99.76, transcribed to every 10000 test takers we have 24 students achieve that range. Or 1.7 mil, we have 4000. This one conflict even more, usually at end range 99+, we have 7600+
2. Also, we also squash historical and traditional cut off, from @123field, records- we never cross lowest 99+
3. You move up so much, but the very top stay the same, no where to go
4. With that assumption, top scorer (perfect scorer) would be around 1200, I was wild to assume from 100 to 550. This is more and more Wild Wild West
5. When you move that much, why the lowest range is still considered reference point?
6. You probably know when students taking old SATs, we have experimental sections for many years with experimental questions – CB use those to give test --> CB has data on their questions already
7. There is no overnight or over one test, we -as students- get smarter because CB has no ideas what to give (remember they did have experimental sections and experimental questions)
8. A little story, my mom (a teacher) monitors her on line exam class in real time. She knows what questions easy or difficult in real time at her monitors - think CB has better computers, better people to monitor compare to my mom
9. CB pondered about these changes many years, they probably have some mistakes but will be insignificant.

Many but will come. But I got headache, mom wants me to put dishes up from dishwasher to the shelves. I got to stop

[dallaspiano]

Mom says I spend too much time at computers, too much time with parents and that I got to stop and go out and play with my friends. Tough mom, o la la I will be out the house in no time next year