(I'm not in the admissions committee or anything; this advice is not Caltech specific)
Your acceptance can be rescinded. The general impression I picked up when I was a senior last year was that if you drop a letter grade in one class, you're almost certainly safe. If you do worse than that, you could be putting yourself at risk. I don't personally know anyone who had their acceptance rescinded, but this was the guideline given to my class by our school's college counselor who had seen people get their acceptance rescinded.
It's okay to slack off in your 2nd semester, but do so within reason.
(note: If you usually get straight A's and then you got B B B C C, that'll probably mean trouble. If you usually get straight B's and then got those grades, I wouldn't worry too much)