OP: In post #4, you list 11 other schools to which you would have applied if you had not been accepted ED to Williams College. Those eleven schools (RPI, Univ. of Florida, Penn State, Emory, Chicago, Cornell, Columbia, Tufts, Vanderbilt, Johns Hopkins & NYU) are substantially different than Williams College in almost every way other than regarding academic excellence.
Colleges similiar to Williams College are Dartmouth College in New Hampshire & Middlebury College in Vermont & Bowdoin College in Maine.
I agree with post #8 above that Winter at Williams College is likely to reveal whether (weather) or not your fears are justified.