Whether it has an impact will depend on your target schools. For schools that recommend or require 4 years and your peers will have completed an AP level, it is less rigorous.
My kids took classes they were not interested in. On a certain level, it’s sort of expected. However, when my son really wanted to drop from IB English senior year to take regular English I was totally supportive. In the end, he switched back to IB because the school had new teacher that the kids seemed to like.