All excellent academically and in terms of prestige, so let’s forget rankings.
None are in urban centers, but Williams is known for being particularly isolated, to the point where it’s a turn-off for some people. there is really no town there at all. If you have any doubts about your ability to thrive in that for 4 years, and could see yourself maybe getting bored, then I wouldn’t take the chance with it when you have such excellent options. If isolation doesn’t bother you, if you prefer a hike on Sundays (every Sunday) rather than brunch and a movie, then maybe Williams is for you. Again, if in doubt, maybe don’t risk it.
So I see your decision as between Williams vs Vassar/Middlebury.
Vassar is not a warm weather place, but Middlebury is quite cold. And it was the most beautiful campus I have seen. Vassar is lovely too so this is not a reason to choose, just my observation.
Vassar typically has more women than men (like 56% to 44%). Middlebury is more balanced. stereotypically Vassar leans artsy and Middlebury has more athletes, but these are just stereotypes and YMMV.
Middlebury really emphasizes foreign language and immersive travel abroad.