If you’re not enthused by the sporty culture, you may want to take Middlebury and Bowdoin off your list and put Vassar on.
I would try to revisit. Smaller schools definitely have vibes, and it sounds like you will know what you’re looking for when you see it. You can explore departments when you’re there (although I would advise looking through a course catalogue before you visit so you know what you want to explore further in person.)