I almost always think that quitting band late in high school is a bad idea, especially if you are good, have a leadership position and/or have the grades, etc. to be looking at very selective schools. Band takes time, fortitude, emotional (and in the case of marching band, physical) strength, and colleges know this.
My kids all pursued band all the way through high school. They didn’t love it very often, most often they just tolerated it, but it was a constant in their lives. It also provided a different way of thinking and “being” for one period during the school day. D1 said she needed that class period as a change of pace each day to re-charge and get her through her otherwise very heavy course load.
D2, who liked band the least, ended up sticking it out through high school and is the only one of the three who played after high school. She’s now in college marching band (she had never marched before - it was a very last minute, shocking decision) and is obsessed with “all things band.” She has a huge network of college friends, including those she will live with next year. She’s even taking up a second instrument and enjoying the challenge of doing so. Sometimes I feel like she’s a total stranger; she did nothing but complain about band in high school.
Lastly, being super busy with productive activities like band helps you to be more organized and budget your time better (because you have to!). If you end up with more time, you could start being wasteful with it. I think that focusing all your energy on pure academics will rob you of some of the more pleasurable and fulfilling things in life. Doesn’t seem worth it to me (and as someone who attends many admissions events as a volunteer for a very selective school, I think you would seem less interesting to admissions having given up band to do yet another AP class…).