I’m also in the it “depends” camp. I think getting an overall feel for big/little, urban/rural can be helpful.
I’d visit UVM while in VT and contrast it with a school like Tufts in Boston.
FWIW, we did start visits between sophomore and junior year. We had friends with older kids who suggested doing that and then sending her to a summer program at her top choice between junior and senior year. It really helped her process and took some stress off doing the visits early. We also did make some repeat visits as she approached senior year.