Williams has truly awesome Math, Physics, and Astrophysics departments. Math profs consistently win national teaching awards (and of course math is closely intertwined with Physics), and the research opportunities in Physica/Astrophysics at Williams, in terms of high-level work without any grad students to compete with for resources, are unparalleled. You simply can not do any better as an undergrad. Check out, for example, Williams’ list of Apker award winners:
http://www.aps.org/programs/honors/awards/apker.cfm
Christopher Chudzicki, Nathan Hodas, Charles Doret, Nathan Gerke, all since 1999 … any only two students per year win these, from every college and university in the country.
This gives you an idea of the types of opportunities you will have as a Williams Physics grad:
http://physics.williams.edu/people/graduates/
The work load, while rigorous, is certainly not overwhelming, especially when compared to Cornell, which is notoriously a fairly stressful place. Dartmouth and Williams re probably similar in this regard (as they are in many regards, the main difference being no frat culture at Williams, a slightly less conservative environment overall, half the size, and no grad students).