It makes some difference, maybe considerable difference, whether you ask this question as a tenured full professor or as a brand-new Assistant Professor. There are few better gigs in the world than having tenure at Harvard, which is famously decentralized, the faculty has lots of power, the teaching loads are light, and the opportunities to monetize one’s prestige to earn extra income are many. Senior faculty move around some, but a lot more seem to move to Harvard than away from it. Being an Assistant Professor at Harvard, Yale, or anywhere of that ilk is much less of a good deal. One of my cousins was an Assistant Professor at Princeton, and it was an absolutely miserable experience. He taught 13 different courses in 6-1/2 years, most of them outside his area of specialization, and was told in the middle that because his department had to shrink he would not be considered for tenure regardless of his record.
Of course, I don’t know where it’s great to be an Assistant Professor. That’s generally a job that looks good only as compared to being a non-ladder Lecturer or Adjunct Professor.
Another place that is uniquely great for senior faculty is All Souls College, Oxford. There are no students, and therefore no teaching responsibilities, although most of the faculty voluntarily supervise some PhD students. There’s also no tenure, however, or very little (there may be a handful of fellows with tenure, but most are on successive 5-year contracts).
Re: Colby. I have a relative who has spent his entire career on the faculty there, with occasional fellowships, leaves, and visits elsewhere. He absolutely adores it, and he has a wonderful life there. He is always ready to explain in meticulous detail exactly what makes it better than Bowdoin or Bates. I wouldn’t go so far as to say he would have turned down an offer to move to Harvard - he has done stints as a visitor there - but it’s hard to imagine he would have been happier or had a better life there.
The vast majority of folks who start out as assistant Prof. at HYP or peer elite Us tend not to stay for their entire careers unless they were not only exceptional hires, but also excelled far above and beyond their fellow peers in research/publications.
Most end up having to leave for another tenure-track position at a lower-tiered university/college. This is more common and if one’s performance as academics is excellent(top of their fields)…especially in their respective fields, they may be invited back laterally as tenured senior Profs in the middle-latter parts of their academic careers.
Also, not all Profs feel the need to take the plum lateral offer when given the opportunity because they’re fine at the lower-ranked institution where they received tenure and built their academic careers.