I think each has a different atmosphere. Caltech is very rigorous and heavy on theory, Stanford's very much about practical/industrial applications, and when I visited them, both said MIT was a blend of the two.
Is the decision for you mostly "Boston or California"? What are you interested in studying, and what are the dimensions of undergrad experience that you're most concerned about or questioning? I'll have more to say once I hear a little more about your interests and concerns.
I chose MIT and never looked back. MIT's great - it's bigger than CalTech and has fantastic people. It's in Boston as opposed to Pasadena, and is just surrounded by so much culture that I for one certainly love it. I'm on the debate team and hang out with people from other awesome New England colleges all the time - I've friends from Harvard, Yale, Brown, etc. I just don't think CalTech could provide that.
Then again, I'm sure CalTech has its own charm. Two of my friends go there; they love it. If you can't decide, I definitely agree with the suggestion to visit both, to see which one is right for you. When you get to schools of MIT and CalTech's caliber, the education you'll be receiving will be first-rate, no matter which you go to. It's all a matter of how much you love being there that's really important - and that really is a very important decision to make, too.
Congratulations! These are both fabulous schools, and you have a wonderful choice. Try to spend time at both (preferably outside of their admitted student week-ends when things are more typical). The right answer for you should become much clearer. The schools' sizes, campuses, surrounding communities, and weather are obvious differences. The schools' cultures and "feel" are also quite different. Here are some random thoughts after visiting both places several times with my son, and after hearing him discuss his visits. (caveat - the visits were short, and I'm not a student. Others may give you more informed impressions)
Caltech, perhaps because of its smaller size, seems more academically focused than MIT. If you are oriented towards research and already have a clear idea of your interests, Caltech might be better. Research and academic focus are available at MIT also, of course. But if you view college as a time to try lots of different things, MIT would give you a broader range of classes and activities to sample.
I don't honestly believe that students at either school are significantly more or less happy or stressed or overworked than at the other. But there seems to be more of a "united in adversity" culture at Caltech. Caltech students are more likely to say "don't go here", "it's too hard", etc. I think it's sort of a Marine mentality, like saying it's only for The Few, The Proud, and mere mortals shouldn't attempt it. MIT students are more likely to say that it's fun and they are having a blast, and to encourage prospective frehmen to attend.
I think the top students are similarly brilliant and hard-working at both schools. There is probably more of a difference in the bottom students. If someone who was not brilliant and hardworking found themselves at one of these schools and just wanted to squeak by, I think that would be much easier to do at MIT than at Caltech. If you got there and decided that you aren't really as passionate or talented as you thought you were in nuclear physics or complexity theory or whatever else you thought you were going to devote your life to, you would have more options at MIT. At Caltech, you might have to transfer out.
The MIT campus has more people on it per sq. inch, particularly when the weather is bad and everyone is inside. This increases the urban feel and sense of energy. Caltech has a gorgeous campus, with much more use of outdoor space and a greater feeling of serenity.
At Caltech, student social life seems centered on the houses which operate sort of like a cross btwn dorms and fraternities. There are not a lot of places outside the houses where students seem to congregate and you don't see a lot of students in common areas. Undergrad houses are close to the classroom buildings, which may encourage people to go back to their house if they have some time off btwn classes. At MIT, people are loyal to their dorms/halls, but also seem more likely to form groups based on interests. Some of the dorms are too far from the classroom buildings to reasonably go back to your room during breaks btwn classes, and there seem to be more people hanging out in public areas.
Everyone knows, I am sure, what is meant here. Most people feel like MIT is in Boston, since Cambridge is more of a neighborhood in the city and less of its own separate town (visit and you'll see what I mean). LA is a completely different ballgame -- I'd almost go so far as to say there is no such thing as the city of Los Angeles, since it's such a huge area of diffuse urbanity with neighborhoods like Pasadena nestled throughout -- many of my profs, who have lived here forever, agree. People don't mean "MIT is in Boston" as a bad thing -- in fact, I think it's a very good thing. When I'm visiting my girlfriend, I love being able to hop on the T and be virtually anywhere in that great city in half an hour. It's not like people are putting down MIT for being, essentially, a part of the city of Boston. It is, in many ways, an excellent feature. Some people wouldn't give it up for all the orange trees and eternal summer in the world.
Go see both places. Is there a theme developing? ;-)
By the way, as a Caltech student who really likes MIT, it makes me happy to see so much good, informative, many-sided and even-keeled advice on both the Caltech and MIT versions of this thread. It's great that people aren't just blindly plugging their own schools but trying to help admits pick the best one in their situations.
I agree that the workload sometimes gets to you at both places, but at Caltech at least you almost always have a sunny, warm day with blue skies to enjoy after handing in a problem set, which I've found lifts the spirit. This is anecdotal, and I'm sure many people disagree, but a student visiting from MIT once said "Gee, you guys seem so much less miserable. Maybe it's that there's no cold."
Neither MIT nor Caltech have that much random-person-on-the-street name recognition (weird, isn't it?). If you want that, go to Harvard or maybe Yale. MIT is probably better known in engineering circles, but the Caltech name probably has somewhat more impact in, say, theoretical physics and astronomy and hard-science chemistry. Certainly, among the people who matter for hiring you or whatnot, MIT and Caltech both come in with sterling reputations as the best math/science/engineering institutions in the country.
Oh, and good point that downtown Boston is closer to Cambridge than downtown LA is to Pasadena, making Pasadena a little more quiet and peaceful-feeling, and MIT more bustling and busy. Which one is better obviously depends completely on your priorities and tastes.