@skipper093 : In your situation, I think that Wesleyan is the safer & wiser choice simply because you are not an athlete & Middlebury is smaller & more isolated than is Wesleyan.
Although Middlebury’s campus is quite beautiful, you will be exposed to a more diverse population at Wesleyan because it is in Middletown.
No wrong choice, however.
Congratulations on being accepted to two outstanding schools !