I may have missed it, but it sounds like you haven’t visited and won’t be able to visit before making a decision. If that’s the case, then you’re in a pickle because Wes and Vassar are probably more alike than any two other colleges in the NESCAC (noting that Vassar is not a NESCAC member). They are alike in culture and, to an extent, in what they’re good at, though each school has its own areas of pronounced excellence.
That being said, they are different places and campuses. Wes is integrated into Middletown, Vassar is walled off from Poughkeepsie. Vassar has a fairly organized layout of pretty old buildings, Wesleyan has a much more eclectic mix of buildings, new and old, and the layout is less classical, though there are many beautiful buildings at Wes. The point about athletic fields in the middle of campus is partially correct. Most of Wes’ athletic fields form part of the athletic complex which adjoins campus. But Andrus field is dead smack in the middle of campus, though outside of football season it’s just a green space; there is no permanent stadium infrastructure. It’s built out during football season. The soccer field is by the CFA, and when not in use is also just another green space. When they’re in use, it’s actually a cool and pleasing collegiate aesthetic, whether you care about the games or not. And Andrus has a long history. Andrus Field 1831–1911: Athletics and the Environment - Connecticut History | a CTHumanities Project.
There is also a slight difference in environment due to Wes being a NESCAC school and therefore emphasizing athletics a bit more than Vassar. If Vassar were to ever join the NESCAC, a conference in which they would be a natural fit IMO, that would likely change. Until then, I think it makes a bit of a difference in campus feel. In that respect, Wes and Amherst are more alike. Football, especially, is a difference maker wherever you find it, and Vassar doesn’t have it (for better or worse, depending on your tastes).
FWIW, Amherst is third place among these schools for campus attractiveness IMO. Though the town of Amherst is more bucolic than either Middletown or Poughkeepsie, I have never been all that impressed by the college’s campus. It’s well put together and nice, but also a little boring in my book. I would put Middletown over both for restaurant options.
I think the study of history would be robust at both schools. If you can figure out which place you’d prefer, go there if Amherst doesn’t work out.