A correction - it's not so much that we felt that Carleton was more "intellectual" than the others (that's prob not fair to the others which all offer plenty on the intellectual front), but more that the level of academic intensity seemed a bit higher. It could be because of the trimester system where things go by a lot faster.
On Grinnell, another aspect is that there does not appear to be a culture that stands out a lot more than others. To look at gross stereotypes (but there's often a reason for these stereotypes), Vassar is often cited for artsy and hipster types and Wesleyan for activists; Carleton has less of a stereotype associated with it - maybe quirkiness. Grinnell has all of the above (although I'm not sure that there are many hipsters - maybe a current student can comment), but does not appear to have a particular subculture that stands out. I think this is part of what allows so many students to feel so comfortable there and in fact rave about their experience - the eight Grinnell alums S talked to in detail were almost cult-like in their enthusiasm (and these were all alums that were personal contacts, not referred by the school).
Another factor to mention is the relative size of the endowments, which can have a significant impact on both your immediate college experience and the long-term staying power and reputation of the college. Grinnell's is one of the very top endowments in the country on a per student basis, and it shows in many ways (amazing facilities, the smallest class sizes, more international students because of generous financial aid, free concerts/events, lots of money for new academic initiatives and student activities). Of the others, I recall Wesleyan's student newspaper The Argus talking about the financial issues the univ was having with their endowment, and the impact it was beginning to have on their staff hiring, etc. Not sure how serious it is, but you may want to check it out. For some more info on endowments by college (scroll down for the per capita endowments after you use the link):
List of U.S. colleges and universities by endowment - Wikipedia, the free encyclopedia
Good luck - you'll have a great education no matter what your decision is.