That’s part of it. However, another large part has to do with family income.
Colleges with lower acceptance rates have wealthier students. The most common reasons for students not finishing within fours years are financial. Moreover, colleges with higher acceptance rates tend to have less resources, and have much less ability to provide support for students who are struggling, whether it be academically or financially. After all, less prepared kids who are accepted to colleges with low acceptance rates based on non-academic factors graduate at pretty much the same rate as students who are much better prepared.