Most common answers on the SAT

fignewton
Mon, 07 Oct 2013 20:17:55 +0000

pckeller:

The screening process should certainly catch problems like I described. But how does it catch them? I believe it is by checking to see that when it appears on an equating section, that the lower scorers are not getting it right at an unexpected rate. But if those lower scorers have been coached to guess C, the screening process will be less likely to catch a flawed problem if it has an answer of A or E.



Hmmm, interesting. Yes, you’re right about how the questions are checked for that kind of flaw. A major source of info for the quality of a question comes from the plot of the percent of students getting the question correct versus their SAT scores on the same test. The best questions will have low percentages for low SAT scores, high percentages for high SAT scores, and a sharp rise in-between.

What’s funny is that stats below are suggestive of fewer A and E easier questions than one might expect normally. Perhaps more than suggestive for E answers.

Here is the distribution for level 1-3 questions:

Total 5-choice questions:   723
Expected number per letter: 144.6
Expected range per letter: 133.8 - 155.4 (one std deviation)

Results:

observed expec one-var z chi-sq
    --------     -----  ---------  ------
A  134  (18.5%)  144.6  z = -0.99  0.777
B  154  (21.3%)  144.6  z =  0.87  0.611
C  155  (21.4%)  144.6  z =  0.97  0.748
D  158  (21.9%)  144.6  z =  1.25  1.242
E  122  (16.9%)  144.6  z = -2.10  3.532   <--


For one variable z, we need |z| > 1.96 for
95% confidence of a non-uniform distribution.

Total chi-squared = 6.91 (4 degrees of freedom)