To be honest, I was always a tad skeptical that there is this sort of one culture per institution that one could put a finger on. But it may very well just be my ignorance talking. I never spent any time on American campuses.
I mean, sure, there certainly would be broad differences between institutions on average, but also two different kids can probably go to the same school and have two completely different cultural experiences, depending on what they are looking for.
Latest events on campuses are a good reminder of that.
I would agree that there has to be a good reason for foregoing both MIT and Harvard from the OP’s list.
The density of the highest-caliber math and physics kids will be highest at those places - if that is a consideration.