As you know, Cobb County released some intriguing data about how their students did, including at the top end of the curve. (References below.)
I did some statistical analysis of that data, trying to use it to determine what the cutoffs would be (just based on total score) if each school were to have the exact same number of NMSF as the previous year.
Some high-level points first:
My results make the cutoffs look VERY HIGH.
These school districts seem to be the same ones that Jed Applerouth advises, so it might be the same datapoint that he presented where scores seemed very high.
I believe this school district / at least many of the schools in it cares a lot about the PSAT (heck, they’re issuing a press release). I think they tailor their curriculum at least in part to increasing students standardized test scores. This might have changed recently. So it’s possible that they have increased the number of NMSF students they’ll have significantly.
The number of NMSF varies a lot from year to year, even in a big school. Walton had 25 2 years ago, and 16 last year. Wheeler went from 14 to 17.
Here is the data my simulation generated. More details appear below it.
Wheeler: With sd 250 top avg is 1378.7 (0.7), with cutoff 1519.5 for 17 / 145
Walton: With sd 174 top avg is 1452.4 (-0.6), with cutoff 1519.1 for 16 / 601
Lassiter: With sd 151 top avg is 1338.6 (0.6), with cutoff 1484.2 for 5 / 423
Pope: With sd 171 top avg is 1317.8 (-0.2), with cutoff 1485.7 for 5 / 361
Allatoona: With sd 139 top avg is 1156.4 (0.4), with cutoff 1396.0 for 1 / 233
Kennesaw: With sd 228 top avg is 1266.0 (0.0), with cutoff 1516.8 for 6 / 135
Harrison: With sd 150 top avg is 1258.5 (-0.5), with cutoff 1478.6 for 1 / 426
Hillgrove: With sd 140 top avg is 1194.8 (-0.2), with cutoff 1440.2 for 1 / 207
North Cobb: With sd 176 top avg is 1174.4 (1.4), with cutoff 1474.6 for 1 / 194
As you can see, projecting the cutoff from Wheeler / Walton data would yield a cutoff of almost a perfect score! Projected cutoffs based on other schools with only 1 NMSF last year were sometimes a lot lower - as low as 1396 for Allatoona.
The way my simulation worked was the following. Assume a normal distribution, with the mean as given. Try many trials with lots of different standard deviations, and see which standard deviation was most likely to generate a “top-100 average” that matched the data given by Cobb. Then, for that sample of data, if the school was expecting 17 NMSF, average the 17th best score with the 18th best score to get the projected cutoff.
Yeah, 1519 cutoffs seem pretty crazy. My guess: “assume a normal distribution” is where the trouble starts. This test was brand new, and the discrimination at the tails that would normally happen didn’t happen. Which also explains why their published percentile tables make no sense when compared to actual results. I would assume that the CB tried to fix it with their scoring tables and couldn’t.
That’s all conjecture on my part. I don’t work for the CB and I don’t do this for a living. Right now I don’t even have any suggestions for changes to the simulation to make it spit out reasonable results.
@thshadow
If you’ve got the Monte Carlo simulation set up and would be willing to play…
(1) As you note, the # of NMSF for Walton last year was relatively low (Georgia’s cut score was at a high relative to other states’) . What would happen to the projected cut if you allowed Walton 25 NMSF this year? What about if you allowed it 50 NMSF, just to get a sense of how dramatic the increase would be with a lower cut?
(2) And going the other way: Using the total score is going to distort upward the number of expected Walton NMSF for a particular cut (though 1519 can’t be distorted too much!), since Walton does appear to excel at math, and verbal has always been double counted for NMSF qualification. Maybe there’s no way to correct for that, given the data we have? 1450 is a score that yields a 216 SI (prepscholar, testmaster) and approximately preserves the ratio of Walton’s top-100 math mean to its top-100 verbal mean (740 math; 710 verbal). What number of Walson NMSF does a 1450 cut suggest?
Yes, assuming a normal distribution is presumably the problem. I also tried a gamma distribution and a half-normal distribution (because those were easy to do with my perl script ), but that only made things worse.
To answer your question @Lea111 - I can also run it and assume that every school gets 50% more NMSF than they did last year. That yields:
Wheeler: With sd 233.594 top avg is 1378.1 (0.1), with cutoff 1497.1 for 26 / 145
Walton: With sd 174.424 top avg is 1453.0 (0.0), with cutoff 1508.8 for 24 / 601
Lassiter: With sd 150.772 top avg is 1337.7 (-0.3), with cutoff 1455.2 for 8 / 423
Pope: With sd 170.323 top avg is 1318.0 (0.0), with cutoff 1455.7 for 8 / 361
Allatoona: With sd 137.373 top avg is 1156.0 (0.0), with cutoff 1359.6 for 2 / 233
Kennesaw: With sd 218.926 top avg is 1266.1 (0.1), with cutoff 1496.0 for 9 / 135
Harrison: With sd 150.498 top avg is 1259.6 (0.6), with cutoff 1453.3 for 2 / 426
Hillgrove: With sd 139.326 top avg is 1194.9 (-0.1), with cutoff 1404.5 for 2 / 207
North Cobb: With sd 173.653 top avg is 1173.1 (0.1), with cutoff 1438.1 for 2 / 194
Not really much of a difference for Wheeler / Walton…
If we double the # of NMSF per school, it yields:
Wheeler: With sd 237.608 top avg is 1377.4 (-0.6), with cutoff 1456.8 for 34 / 145
Walton: With sd 174.102 top avg is 1453.2 (0.2), with cutoff 1488.3 for 32 / 601
Lassiter: With sd 151.250 top avg is 1337.7 (-0.3), with cutoff 1441.4 for 10 / 423
Pope: With sd 170.430 top avg is 1317.7 (-0.3), with cutoff 1440.0 for 10 / 361
Allatoona: With sd 138.125 top avg is 1155.8 (-0.2), with cutoff 1358.8 for 2 / 233
Kennesaw: With sd 218.301 top avg is 1265.8 (-0.2), with cutoff 1469.1 for 12 / 135
Harrison: With sd 150.039 top avg is 1259.1 (0.1), with cutoff 1451.3 for 2 / 426
Hillgrove: With sd 139.307 top avg is 1194.9 (-0.1), with cutoff 1403.2 for 2 / 207
North Cobb: With sd 173.653 top avg is 1172.8 (-0.2), with cutoff 1437.8 for 2 / 194
I’ll add one more piece of data to this, though I’ll note
this is veering further and further from any kind of scientific prediction
it’s pretty stupid that I’m using various techniques to try to estimate what the 20th highest score is at a certain school. Anyone that sees the actual data could just announce it! This is not what you use statistics for.
Anyway, if you assume that Wheeler and Walton each have 50% more NMSF, and furthermore that their scores are bimodally distributed (I just took 2 normal curves with the same variance with a fixed mean difference and added them together), you can get cutoff scores as low as around 1450.
Wheeler smartFrac 0.6, meanBoost 300: With sd 92.510 top avg is 1378.3 (0.3), with cutoff 1454.2 for 26 / 145
Walton smartFrac 0.2, meanBoost 300: With sd 22.579 top avg is 1453.0 (0.0), with cutoff 1465.7 for 24 / 601
The brief summary is, if I had to guess, the Cobb county data supports the concordance table view, and not the percentile table view.
@thshadow Thank you so much for doing this! I really appreciate being able to ‘make/speculate some sense’ of the Cobb county, and in particular Wheeler data. It is very interesting that your analysis gives a nod to the concordance tables. Thanks again!
Nice work. I have a question, or rather some observations that maybe you could make sense of since you seem to be very well versed in statistics.
First of all, I wish they would’ve published the median scores for the top 100. That would’ve been very informative. If I were a GA resident and I saw these numbers I’d be thinking 212-214 or so for SF. The reason is that I believe that is that I believe that a big chunk of Walton’s SF scores will be ridiculously high like 222-228 because of their high math average. This actually may be a “career” year for them as far as SF in terms of total SF.
But looking at the data I noted that a school that compares favorably to them (# of SF last year) is Wheeler. Last year Wheeler had 17 SF while Walton had only 16. This year Wheeler’s top 100 averaged 205.6 that’s 11 points lower than Walton’s 216.6. That difference is pretty significant. Wheeler to me appears like a representative sample of a good, quality school in GA while Walton appears “Roided up” if you will. To account for the average disparity do you think Wheeler has a pocket of 20-30 or so kids at AP level that scores disproportionately high compared to the rest of their test takers and there is a sharp drop after #30 or something along those lines? Perhaps Walton just has a ton of kids scoring right at and below the cutoff and hoarding up commendeds. Walton, at least going by the eye test, looks like a Super-School that should crank out SFs but last year they had 1 less than Wheeler? It just seems a little odd.
For background, Wheeler has a STEM academy. All Cobb students can apply. Special program. Been around awhile. So, imagine a lot of the high scorers come from this group. Without this program, those kids would be trickled in to their zoned Cobb HS.
Walton also has a somewhat newly formed STEM program for its students. It is very intense and comes with peer tutoring, etc. But, Walton has always been the premiere Cobb HS. Johns Creek in Gwinnett and a few others. And, don’t forget some of the private schools in GA. They will have a lot of high scorers, as well.
@MotherOfDragons Thanks. By “roided up” I wasn’t implying anything nefarious like cheating or fudging numbers, just noting that such a large gap in the top 100 average scores between them and wheeler (a school who actually produced 1 more SF last year than Walton) was striking. You normally don’t see that many consistently great scores throughout the Top 100 in very many schools. As I mentioned, you’ll usually start seeing at least somewhat of drop off after the AP, Honors, etc. That likely isn’t the case at Walton. It looks as if they have a ton of bright kids with no drop off in sight.
I’m not from the area and did not research it. I just looked at the numbers based on two pieces of data. In hindsight I realize that “roided up” has a negative ring to it. “Exceptionally strong” would’ve been a better choice.
Whether any of the schools are “roided up” or not , what really matters is if they’re going to have significantly more NMSF than last year. It’s certainly possible to believe that they had more smart kids this year, and/or the school spent more time preparing for it than in the past. But Applerouth has stated that he’s seen lots of other schools all around the country which would have way more NMSF if - IIRC - you add 12 to everyone’s score. I think he’s arguing (a) you can’t add 12 to your SI, and (b) predicting NMSF by percentiles is also overly optimistic.
@mozart6023 - yes, my bimodal distributions were attempting to model something like that, where there are a separate group of “smart” kids. There are definitely distributions (and plausible cutoffs) where the lowest TS to get a NMSF at Walton / Wheeler will be in the mid 1400’s. But that doesn’t prove anything at all.
All I can say is that it’s easy to see why Applerouth - who presumably has seen more detail at least for these schools than has been released - seriously doubts that the NMSF cutoffs will correspond to the percentiles. The percentiles predict, what, a 210 would be NMSF in GA? It would probably causes these schools to have 50 - 80 NMSF each, which is probably crazy…
@theshadow It’s impossible to tell but fun to guess. Anything is possible I guess without more data. As I said earlier, a median and/or mode would tell so much more but a 216.6 average over a 100 scores? That is impressive.
Projected cutoffs based on other schools with only 1 NMSF last year were sometimes a lot lower - as low as 1396 for Allatoona.
), but that only made things worse.