It is unlikely to be a problem when applying to a PhD program, which will holistically look at your record, with greater emphasis on upper level courses in your major than on out-of-major courses taken while in high school.
However, medical and law schools will include all college grades into the GPA that they calculate.