<p>The screening process should certainly catch problems like I described. But how does it catch them? I believe it is by checking to see that when it appears on an equating section, that the lower scorers are not getting it right at an unexpected rate. But if those lower scorers have been coached to guess C, the screening process will be less likely to catch a flawed problem if it has an answer of A or E. </p>
<p>So I guess if I had to guess randomly on a hard question, I would go with A or E. But, of course, this is all just silliness: no one has to guess randomly on ANY question! Devising clever guessing strategies is, for the most part, a distracting waste of time.</p>
<p>I wonder what Fig’s data would look like if dataset 1 was done on only difficulty 1-3.</p>
<p>It sounds like A,E would be worst choice in general, but the best for high difficulty. The advantage would be more pronounced if the datasets were split 1-3 and 4-5. It might be an 8% advantage.</p>
<p>Hmmm, interesting. Yes, you’re right about how the questions are checked for that kind of flaw. A major source of info for the quality of a question comes from the plot of the percent of students getting the question correct versus their SAT scores on the same test. The best questions will have low percentages for low SAT scores, high percentages for high SAT scores, and a sharp rise in-between.</p>
<p>What’s funny is that stats below are suggestive of fewer A and E easier questions than one might expect normally. Perhaps more than suggestive for E answers.</p>
<p>Here is the distribution for level 1-3 questions:
Total 5-choice questions: 723
Expected number per letter: 144.6
Expected range per letter: 133.8 - 155.4 (one std deviation)</p>
<p>Results:</p>
<pre><code> observed expec one-var z chi-sq
-------- ----- --------- ------
A 134 (18.5%) 144.6 z = -0.99 0.777
B 154 (21.3%) 144.6 z = 0.87 0.611
C 155 (21.4%) 144.6 z = 0.97 0.748
D 158 (21.9%) 144.6 z = 1.25 1.242
E 122 (16.9%) 144.6 z = -2.10 3.532 <--
</code></pre>
<p>For one variable z, we need |z| > 1.96 for
95% confidence of a non-uniform distribution.</p>
<p>Total chi-squared = 6.91 (4 degrees of freedom)