This article from the Williams Record on students who transferred out of Williams will offer you an insider’s perspective on the types of students who are not fully satisfied there. The article notes that
It seems that, for your criteria, Swarthmore’s proximity to historic East Coast cities could represent a big draw.
Nonetheless, there are aspects of Williams that strongly appeal to you. And, as a personal comment, Williams offers the broader curriculum overall, which would appeal to me.
For these reasons, I hope you continue to consider both of these schools, and attain a decided preference for one of them.