Physics 1 is the most difficult AP out there - it has the lowest number of students who get scores of 5 of all AP test. In a large part this is because it isn’t easy to make sense of physics if you use algebra, as opposed to using calculus. So learning physics using only algebra is difficult.
This means two things. A, you may need more algebra training than Algebra 2 to be able to use algebra for physics, and B, not doing well on Physics 1 does not mean that you will have a difficult time with physics in college, since physics is taught in college using calculus.
On the other hand, if you have a difficult time getting at least a B on Calc BC, you may find physics extremely challenging.