IzzyD, I can tell you my son's experience. He isn't into drinking and drugging, and he had some concerns his first term about finding like-minded people, about his choices being respected, etc. He's found a bunch of great friends who aren't into a big party scene. You can find that at Carleton, but you can also find its opposite, and neither choice is judged. I think the one thing that *is* looked down on at Carleton is looking down on others' choices.
You can do the old trick of putting both schools' names into a hat and picking that way. Then, listen to your feelings when you draw out one name--you'll know if you really want to go to that school or not.