Of course it’s possible- but it’s easier at colleges/universities that have more flexible GenEd requirements (such as Brown, Vassar, Bard, Beloit, URochester, etc). If you have a state flagship or other large universities you already know that you are interested in, check their fine print and see how many of the GenEds you can take during high school - with those out of the way it will be easier to focus on your major subjects. Take your math as far as you can while in high school- that helps a lot with physics.
But: don’t get over focused on your major just now. You don’t have to ‘buy’ yourself more time: you have the time: almost 5 YEARS before you will have to declare. You will have changed as much between now and then as you have between 4th grade and now (which I have to assume is quite a lot!). So, go ahead and the math classes and putz around with gen ed requirements- it can’t hurt- but don’t get too inflexible.