The issue here is that students for whom English is not a first language may well be able to read a psychology or composition textbook fine, but wouldn’t make heads nor tails of Ethan Fromme. Literary texts from the 19th century, 18th century political philosophy texts… will be completely off for them. They won’t be able to show what they know because the texts won’t be appropriate for their level of language mastery. They may speak and read very well, even long texts, but written in contemporary language.
Perhaps ELL’s will be allowed to replace the SAT with the TOEFL and Math1/2 then, but as of now this isn’t on the table.
Poor students tend to attend lower-performing schools, where a significant part of class time is spent on explicit meaning (plus classroom management depending on the class), except for those who have access to APUSH or AP Lit, and that would be a tiny minority. They may be bright but they’ll be tested on skills they’ve never been taught and haven’t been able to practice.