Students don't really live at Yale over the summer - the University uses dorm space for programs and events, as well as conducting remodeling and heavier maintenance during that time. Students do stay in New Haven when conducting research, doing internships, etc. but typically find an apartment for that time period.
During Thanksgiving break, the campus becomes a ghost town - by noon the sunday after the game this year probably 95% of campus had vacated. Some international students even go home (typically those in Canada/Latin America/W. Europe). One thing to bear in mind about breaks at Yale is that the dining halls close during this period. That makes it particularly difficult to stick around. It was, in fact, a rather big deal that one dining hall was open for a few days over thanksgiving break. If you are intent on staying over thanksgiving, I think most colleges do have some sort of thanksgiving dinner.
I think spring break tends to have more people around. Seniors have stuff going on, people on sports teams are still in their seasons, and so that feeds back into more activity taking place on campus for those two weeks.
I will wildly speculate that fall break this year will be more lively on campus than the other two, just because it seems unlikely that anyone who lives outside the Northeast will find it practical to go home during that period.