Second half of manifesto incoming…
@thshadow I’ve enjoyed running things past you so far, so let’s continue. I’d like you to help me with something I’m struggling with. We’ve discussed a lot of this before. Let’s use Georgia again.
Let’s say we are looking for some way to generate an SI table for this year that has GA’s cutoff at 216. That doesn’t seem to be an unreasonable number based on the anecdotes we’ve seen.
We know GA is typically in the 3rd or 4th from the bottom of the 99s, which in this year’s table is roughly a 207. Somehow we need to concord a 207 to a 216. We can’t just do it because we want to - we need valid reasons to shift the tables.
Note, however, that’s it’s not just us that need to do that. CB needs to do it as well, because their own concordance tables suggest that a 218 last year corresponds to a 216 this year, not to a 207.
We have a few tools at our disposal to do that. One of them is the change in percentile definition. You and I both agree that this change allows us to shift the rank by only one. So this allows us to concord a 207 to a 208. That’s it.
The population that the SI table refers to appears to be a gold mine. But first of all, we need to decide what population the table refers to. If the table is “test takers”, it helps us a little bit. This year roughly 1.72 million juniors took the test. Last year it was 1.57 million. So the population increased by almost 10%. That helps us a little - to be in the top 16000 last year required a student to be in the 90.0% bracket; this year they need to be in the 90.1% bracket. So that would allow me to concord roughly one point higher in the table, so when accompanied by the percentile change, my 207 could be a 209.
If, however, the SI table refers to the “national” sample, then how much I gain depends on my assumptions of where the students fall in the distribution. Publishing a “national” SI table seems silly, but let’s go there anyway. If the phantom “non test takers” perform identically to the test takers, then the “national” and the “user” tables are identical, and I get no help at all - I can’t move up in the table.
If, on the other hand, the phantoms perform poorly, and none are in the top of the distribution, that helps me a lot. Half of the 99+s become 99s, the 99s become 98s, etc. I’m well on my way to getting the results that I want.
Unfortunately for me, and as you’ve pointed out before, I can’t do this. Why? Because we have the “national” and “user” tables for total score, and they are very similar at the top. Not identical, but close. CB has decided the phantoms perform just slightly worse than the test takers. Thus, the differences in population might allow me to move up a couple of ranks. So now my 207 could become a 210, maybe 211. But nowhere near 216.
So here we are. We can legitimately concord a 207 to a 209 (assuming SI refers to test takers), and maybe to a 210-211 if we assume SI refers to national. We’re not close yet - we can’t get to 216.
What’s left? Applerouth talked about a few possibilities. Maybe the sample population wasn’t representative. That seems unlikely though - they sampled 80000 kids. They do this all the time, and they do it for a living. But it’s a possibility.
Maybe they didn’t account for changes to number of questions, guessing penalty, test difficulty, etc. But CB scored the test. They knew the distribution they wanted to target. They could have changed the scoring tables to fit the distribution they wanted.
Maybe I’m wrong about where the GA cutoff should be, and additionally the concordance tables are way off in the same direction my “sniff test” meter is taking me. This would mean GA’s cutoff is around 208, and all the anecdotes across the country turn out to be red herrings. When the final concordance tables come out, the percentiles will fall substantially. But that seems pretty unlikely to me.
So now I’m stuck. I have no intellectually honest ways to concord this years SI table to CB’s own concorded values, or values that seem reasonable to me. Any suggestions?