Although I never attended Harvard one of my siblings did and I would visit him frequently while I was at Penn. Based on my experience I found the Harvard social scene to be extremely dull / depressing which was sad to see given the vibrant social / college scene in the Boston area. It seemed that in general on Friday and Saturday nights everyone stayed in their dorms and didn't really do anything interesting (i.e. just watched TV, read, played boardgames, etc.)...I got the feeling that the place was too "academic." It seemed that the only people that really went out and had a good time were a few of the wealthier international students or those that had lots of friends at places like BC, Babson, or BU. Just to set the context, I'm not advocating a social scene where everyone goes out to crazy parties and gets tanked Thursday through Sunday (a la Arizona State, Florida State, etc.). However, when I contrast what I experienced at Harvard with the social scene at Penn or at schools where friends attended (i.e. Georgetown, NYU, UVA, Columbia, Dartmouth, etc.) I found it to be very limited and the primary driver of this seemed to be the relatively introverted student body as opposed to a lack of social outlets. So if you are a type of person, like myself, that likes to "work hard" but also "play hard" (which includes going out to lots of restaurants with a big group of friends, or grabbing a beer and watching a game at a pub, or going out to a lounge / nightclub, etc.) then you may find yourself kind of bored at Harvard.
Another thing I'd point out, based solely on anecdotal evidence / personal observation (so take it for what its worth) is that I did get the sense that many Harvard students seemed really depressed (not in a clinical sense). When I mentioned this to my brother he noted that he believed that this was due to the fact that many students decide to matriculate at Harvard due solely to the name / prestige as opposed to other factors which if you are choosing a school solely for that then it may not be a good fit and you are are likely to not have that great of an experience. So if you are choosing to attend Harvard solely because of the name then Harvard may not be for you!
On a side note, my brother greatly enjoyed his time there and it was the right fit for him.