I would not seriously worry too much! SPQR08 pretty much has it on target in terms of what Colgate has offer. I'm a stay-in movie type of a gal and I'm okay with it. I live in a sorority house where the girls love to hit up the local bar for happy hour on Fridays and go over to a frat house for a mixer several times a semester.
Colgate is NOT what you see on Animal House. I just survived Spring Party Weekend which is really notorious but it's luster has worn off several years ago when the administration took over the Greek houses. A lot of excitement about drinking comes from the freshmen. It starts leveling off in sophomore year and by junior year, people just couldn't care less about how often they drink per week.
Colgate students know how to play and work hard- literally. True, professors find freshmen obnoxious, especially on Tuesdays and Thursdays (after Freshmen Mondays and Colgate "Thursdays" on Wednesdays) but they're generally good about it. You will definitely find yourself wondering how is it that your classmates who partied the night before managed to come to classes but look a little closer, especially in morning classes, and you'll figure out who tends to party a little too hard
The other factor that will give an isolating feeling is the fact that freshmen dorms are ON THE HILL so freshmen have to wander down to frat houses and the Jug. So once everyone leaves, it can feel pretty weird... but once you're down the hill as a sophomore, you'll be in better position.