You can go, but know that it really won't be an accurate picture of Wesleyan. It's way more lively during the year, and even during most parts of the summer there'll be a lot more going on than there will be this week (students doing summer research often haven't started yet, etc.)
Still, if you just want to see the physical campus itself, go for it. I love the campus itself, but after 4 awesome years there, I'm probably a little bit biased