“yes, but the quality of the kids also matter, so how do you control for that?”
The inputs matter a LOT.
Kids who can get into Harvard or Yale will tend to do pretty well. Regardless of what their Harvard/Yale experience is. And regardless of whether they attend Harvard/Yale.