Colgate and Pitzer are both excellent top liberal arts colleges. Yes, technically Colgate is ranked above Pitzer, but most employers and graduate schools won’t know or care about the absolute numerical rankings of these schools, and they’re not going to run to USNWR to check. They’re more concerned about general reputation of the schools, and Colgate and Pitzer both have similar general reputations. You’ll get a great education and have excellent post-college opportunities from either.
So select based upon where you want to go to school.
I strongly disagree with the advice that location doesn’t matter (and not to make your decision based on that). Of course location matters, and of course you will have time to enjoy it - most college students have a lot of free time. Some colleges/universities pride themselves on their location and integrate themselves tightly into the city in which they are located - both on the urban and the rural side. For example, Columbia takes very seriously the “in the City of New York” part of their name; professors assign projects that involve working with the city, many students volunteer and intern in the city during the term year and a lot of social events (especially in the upper-class years) happen off-campus in the city. Another example - State College, PA is a town that’s all about Penn State, and the small core of downtown plus a lot of the apartment complexes revolve around college life. Campus culture is intimately tied into the life of the town.
I do agree, however, that if you always want to leave campus then you have probably made the wrong choice, and that a large city won’t compensate for a campus you don’t like. I will also say that you shouldn’t discount going to college in a small town. Small towns can be very, very cute and a lot of fun! College students have a way of making their own fun, and there can be something very bonding and exciting about everything in a small town revolving around the school and focused on the student experience. Besides, small college towns usually build businesses around what students need - I’ve been to both State College, PA (where Penn State is; lived there for a year) and New Paltz, NY (friends went to SUNY New Paltz) and they have lots of coffee shops, hangout spots, bars and nightclubs and other businesses that cater towards college students.