I think the biggest difference that might matter is the feeling of the community. While that may be the result of size and location, it’s what is most likely to influence how you feel about being there every day.
Most underclassmen at Tufts venture off campus infrequently. Upperclassmen tend to make much greater use of the city. I think the Amherst community may feel tighter and more cohesive as a result. To some, that clubbiness is a good thing. To others, not so much. You really need to decide which feels right for you.