My SIL is professor of quantum computing. So I hope I am answer some of your questions about the field.
Yes, physics is very much necessary for quantum computing. Math would be a decent second choice. Knowing some CS is useful but not critical.
Are you more interested in the hardware side of QC or the software side of QC because the graduate pathways are pretty different.
Hardware is still in the developmental stage with 4 or 5 competing designs for quantum storage and circuitry. There will be a shake-out sooner or later, but for right now there is no clear “best” method.
Software is also in the development stage because some types of programming are hardware design dependent. Software generally falls into 3 categories: error correction, scalability and quantum-specific algorithm development. All three are highly math intensive.
Few of SIL’s grad students have quantum computing courses from undergrad and most have no or only a limited amount of computer science.