You need three good references for a PhD program. If you skip too much college due to APs, it can be hard to get 3 great references. Those profs can also provide good guidance in the grad school admissions process, which is quite different from undergrad admissions and can be quite opaque.
More research experience is better for grad school admissions. Not all undergrad schools are equal in what is avail to undergrads for research. Again, a student who rushed through may have less experience than other grad school applicants.
Some areas of graduate study require subject test GREs. You want to go to a rigorous enough school to prepare you well for those tests.
You might not get a better undergrad GPA. If teaching quality is poor or lots of sections are taught by TAs that don’t have great English skills or class sizes are really large, it can be harder to master the material for good grades.