Will a BA in physics make getting into grad school more difficult?

[MMRose]

I would check with your advisor and any faculty with whom you are close and share your concerns. Perhaps there are some additional courses they might recommend. Most likely they will tell you that students who graduate from you college with a BA in physics do just fine in grad school applicaitons.