If by graduate school, you mean PhD program, you do want to go to a college with good undergraduate preparation for that PhD program, which is not necessarily the most admissions-competitive college you can get into.
Good undergraduate preparation includes the expected course work (particularly relevant upper division courses of the expected depth and rigor) and undergraduate research opportunities.
Pathways to professional schools (e.g. law and medical) are somewhat different.