How much will a low grade in junior year hurt me? COVID-19 affected my grades.

I take the IB in an international school, and I received a 5 for Higher Level Math (equivalent to a B in AP) in the second semester of my junior year. I’ve received a 6 in Math for every other semester before junior year second sem in high school; COVID-19 really messed up this grade, as the grading system had to be adjusted.

IB - AP
7 - A+
6 - A
5 - B
4 - C
3 - D

I want to major in physics in college, and I take Higher Level Physics, which I scored a 7 on. My last Higher Level course is Economics, which I also scored a 7 on. My 6 courses added up equal to a score of 36.

I’m genuinely very worried about this math grade. For my intended major of physics, I feel that colleges would focus on my math grade a lot, and the 5 in junior second sem would greatly affect my application. I know that junior second sem is the “most important” for college applications, so I’m not sure how much it matters that I’ve previously always received a 6. I planned on applying to the best colleges in the US; I hope to attend a top 20 school. How much should I be worried? Will this be a dealbreaker for an intended physics major in college applications?

For additional info, I have a 1500 SAT (780 math, 720 EBRW) which I plan to retake. I also have a 800 in the SAT Math Level 2 subject test, and a 780 in SAT Physics subject test.
I also attend a very competitive private school where basically every student attends multiple academies and tutors outside of school to improve their grades. I do not attend any of these, which does give me a disadvantage in terms of grades.