It depends on the college and maybe your math grades so far. There likely isn’t anything you can do about it now (you could have looked for a way to take pre-calc this summer, but that is past). A few things:
- There are a few colleges that won’t accept you at all without Calc – Harvey Mudd, for example. But only a small number of top colleges would do that.
- I wouldn’t say you are fully committed to a math major without at least having taken Calc – who knows if you even like higher level math? But the good news is that colleges don’t care a ton what you plan to major in. Many students change their mind.
- If you are shooting for the very top colleges, you likely will be hurt by not having taken the most rigorous course offerings at your HS. But most colleges will be okay with it.
I can’t speak to the Physics class situation. In general it is a bad idea to take a math-heavy Physics course without having completed the pre-requisite math. Do the other students have better math grades or a previous Physics course taken?