It depends on the entire package. If you have a good reason to show why you did not take Physics till the senior year, it will not be a factor if your records indicates that you will do well in the class.
Otherwise, they could defer the decision and then you have a chance to update them with your grades and you will considered in regular decision.
So if you have concerns, it might be better to wait and apply for RD and show the 1st semester grades. Physics is important to Caltech admissions, but not having the grades at the time of application is not a dis-qualifier. You need to look at your package and decide if you want to take the chance with EA (and risk rejection) or wait until RD where you have chance to show the physics grades and appear a little stronger.