For academic careers, it is your PhD school that matters, in terms of how well reputed and prestigious your physics department and PhD supervisor (not so much the entire school) are. But be aware that getting a tenure track academic job is highly competitive, which is why many physics PhD graduates move into other job areas like computing, some kinds of engineering basic research, finance, teaching physics, etc..
Your undergraduate school may matter mostly in terms of whether the physics department produces BA/BS graduates who are seen by the physics PhD programs as capable. I don’t know enough about how much this matters for physics specifically (it matters more for some subjects and less for others).