Calculus is IMHO a bad thing to self-study. The biggest issue I see is that calculus is very, very useful for a lot of other classes that you might take in the future. The only students who might be able to get away with self-studying calculus are ones who are very strong at mathematics, and these are specifically the students who are most likely to use calculus many times. Personally I have used calculus in many classes that I took after I took calculus, mostly including physics and math classes, and I have also used calculus on the job multiple times.
I think that you should wait on both calculus and calculus-based physics. You can learn this as a student in university.