I’ve had a similar thing happen with me while visiting colleges (and I do recall doing the same thing with the Hun School back in the day); sometimes, the tour isn’t even bad, you just don’t feel the vibes of the place. I wouldn’t read too much into it; just go with your gut. If something makes you feel uncomfortable now—no matter how trivial it is—it probably won’t stop doing so in the future, and either way, you’ll still have a bad taste left in your mouth.
Good luck on your child’s applications!