Well, of course. This is college, they don’t have “adults” chaperoning their students. College visits are about letting the student get a feel of what it is like to be a student at the college to see if they fit in.
But obviously as the parent these decisions are up to you, but I would question whether it is wise to not let them go on college visits because the independence of it scares you.