To piggyback on this, take a look at some of the top producers for Physics Ph.D’s: Top Feeders to Ph.D. Programs
The big universities produce a lot of grads in absolute numbers, but there are a number of smaller schools that punch well above their USNWR rankings in their physics departments.
Good Luck to you OP!