My two cents is in many cases, this fades as a huge concern after the first couple years, when you start getting into the advanced classes which are typically much smaller, both students and instructors may be more engaged on average, and so on.
And then sometimes, honors programs can help at least somewhat with the first couple years. Sometimes.
That being said, even if all that is true, if you can comfortably afford the educational experience you want at a smaller or midsize college, either with aid or just because you family has the means, then that is obviously fine.
I just think with all these things, there is more than one path you can potentially take, and at least get a lot of what you want.