I think you are underestimating how much it matters the caliber of teachers, GCs etc who are helping students prepare.
And I would say that schools definitely want to be able to claim having students from as many states, countries as possible - it was a part of the discussion of a few campus tours we attended. Not Ivies, but top 40 schools.