I’m not a student at Amherst College, but as a prospective student who has visited Amherst several times, I’ve been able to get a feel for the Campus and the surrounding area.
As someone from a big city, the town felt very small, almost stiflingly so. While there are many things to do, it remains a small town where stores close early, and where there isn’t a lot going on, especially when it gets cold. For me that was a deal breaker, but many will enjoy size of the town and what it has to offer despite it’s size and location. I’d recommend visiting, and getting an idea for yourself.
With respect to the 5 college consortium, there is movement between the 5 member colleges, but the majority of your time will be spent on your home campus. For that reason, I wouldn’t necessarily count on other colleges to be your main source of things to do.