I don’t think it’s too early, though I wouldn’t take away from some other part of the vacation to do it.
At this age, narrowing down schools doesn’t matter at all. These visits are only to get a feel of whether your child might like big, small, rural, urban, etc. Getting some exposure now when the stakes are low can help them create a list in a couple of years.
The added bonus is that your student can have some demonstrated interest in the books in the event that s/he decides to apply to any of the visited schools who track it. Even if you are not taking an official tour, you can register at the admissions office.
We were grateful to have made some fun visits when my kids were younger because opportunities were curtailed so much during Covid.