This is true.
I just looked at some of the recent student bios from a graduate program (MS) that is very competitive and has a grueling admissions process. Students came from a variety of schools: University of Rochester, Northern Arizona University, University of Pennsylvania, Brown, TCNJ, Minnesota, Pitt, U of Maryland, UCLA, Cornell, Western Carolina, Rutgers etc.
What really struck me was what they accomplished in college and in the year or two following. That was on them.
With that in mind, choose a college that is a good fit for you, and where you would be happy for 4 years (while keeping grad school costs in mind and doing what works for your family).