Current student here. I just finished my freshman year, and my experience was pretty great overall. I would classify myself as "quieter and quirkier": I didn't go to parties often (held at the "Socials"), but my close-knit group of friends and I would have fun doing our own thing. These friends were people from my dorm--I don't know how Amherst did it, but I met nearly all of my closest friends in my dorm. Freshman orientation also provided an opportunity to meet friends, as well as participation in clubs and especially athletics.
Athlete groups tend to be somewhat insular and "dominate" the party scene in that they're generally the ones throwing and attending parties. A capella groups, the Black Student Union, and other cultural groups also throw parties, but it's definitely possible to have a fulfilling social experience outside the context of any party scene.
Amherst (the town) is a place we go to eat out or shop. A great resource Amherst has is the bird sanctuary/forest where my friends and I will regularly exercise/go hiking. It is also pretty easy to visit friends in Boston/New York--Megabus trips to New York can cost as low as $1 (though $4 is the lowest I've gotten myself).
All in all, Amherst is a pretty easy place to be, socially. Though there are certainly cliques facilitated by the prominence athletics, it's easy to make friends and have a great experience as a non-athlete.