It occurred to me that perhaps my own experience as a nontraditional law school student might be somewhat relevant.
I went to law school after a long period in grad school (and also a year off in the middle to write terrible screen plays). When I started law school I was 30, about to turn 31, so a similar sort of age gap with most of my classmates. And in fact I did not end up socializing with most of them–honestly I was sort of amused the degree to which a lot of them almost seemed to be regressing to first-year college sort of behavior. So there was lots of partying, lots of hooking up and other relationship drama, and so on.
There were, however, a few other nontraditional students, and so we hung out together and formed study groups and such. But it was just a handful, and our idea of fun was more to go out for dim sum on a Sunday, not hit the clubs on a Wednesday. I was also married by then, and my wife visited on weekends, so frankly that was most of my social life. And I was also just really busy with doing well in school, such that my weeks were very structured, which in turn meant I could have free time with my wife during the weekends.
And as a result I had a great academic experience–I generally got very good grades, I had great interactions with my professors and largely developed a very good impression among them, got various competitive titles, summer jobs, and prizes (including a notable graduating prize the faculty voted on, and a very competitive summer position that depended on a faculty recommendation), the clerkship I wanted, the Department of Justice job I wanted . . . .
And honestly, it sort of felt unfair I was this focused, organized, mature adult competing with a bunch of distracted kids. But life is not fair, and I knew what I wanted (and didn’t want) out of a legal career by then, so I made sure I got what I wanted.
So this is probably part of why I am personally confident it would be fine going to Amherst. No, you would not likely be a big part of the social scene at Amherst, although again I think you would have some other nontraditional students there, including in your dorm if you chose to live on campus.
But regardless, my view is that for someone who wants to go on to grad programs, this is a feature and not a bug. You, like me, very likely could easily have the inside track to developing professor relationships, to doing great school work, and to generally being the sort of outstanding student who gets the grades and recommendations usually necessary for admission to top grad programs.
And conversely, when other posters suggest there will be more people like you at a Yale or Brown or whatever in terms of grad students and such–yes, and to be very blunt about it, that to me sounds like competition! Like, now instead of standing out as a very rare opportunity for a professor to have an unusually mature social and intellectual relationship with a student, you will be interacting with professors who get that opportunity all the time through their own PhD programs and such.
And I have seen that too, by the way, meaning I don’t think most of the professors in my department when I was a PhD student had much bandwidth available for undergrads. And of course some of those undergrads had ME as their professor!
OK, so obviously you are getting two very different perspectives here. And you will in fact have to choose for yourself what makes sense to you, as I doubt the posters here are going to reach any sort of consensus among ourselves.
But I just feel like having done something similar myself, and having seen the benefits I got, you should at least think carefully about why really standing out in terms of maturity among the students in your institution might not be ideal if you are looking for a great party scene, but might well be perfect if you are looking to college as a way to advance your grad school ambitions.