Physics is usually the hardest major, not only in HS, but also in colleges. Excellent math skills are necessary but not sufficient. Physics requires a level of physical intuition, and even counter-intuition, that is not based on math skills. So, not doing well in physics doesn’t mean you won’t do well in math. If you like math, you can major in math, applied math, or a number of other math-heavy majors such as CS, econ/econometrics, etc.