You should insist on the visits.
I thought my daughter would love a small college campus, and she didn’t care for any of the small schools we visited. She ended up preferring a couple of bigger schools that, honestly, I took her to visit because they were on the way to the smaller schools.
So I’m glad I took her to those latter schools, as well - we never would have known her preferences if I had not insisted on visiting ANY of them. She was also reluctant to visit schools her junior year…