These schools are very different. Duke is the major outlier, and Davidson and W&L are much more similar to Duke (just smaller) than Rhodes and Sewanee.
Duke is a major research university with the med school, law and business schools adjacent to campus as well as multiple phD programs, with an undergrad population of 7000, but also grad student classes available to upper-level undergrads, and grad students involved in the same research labs as undergrads(which is a positive feature, in my experience). The other schools are all much smaller. However, the intellectual vibe at Duke is definitely present at Davidson and W&L, and as the above poster said, there is a bit more diversity and progressiveness at Davidson(and Duke) vs W&L. The academic selectivity of Duke is higher than Davidson and W&L, but not significantly, and there is a lot of overlap of student bodies based on CDS stats.
Sewanee and Rhodes pull from a much different pool academically, with common data sets middle 50%SAT (pre-test optional) 1200-1400 roughly. The kids we know there are great, just not the same type of intellectual, driven kids that attend the other three.
So, these differences may not matter to some students, but these schools have fairly dissimilar vibes.