Should I drop my physics course?

You should go to office hours and discuss this with your professor.