Duke is definitely more into sports (which is fun and great for school spirit), has a much more active Greek scene, and is probably less comfortable for a quirky student. The vibes are so different.
Students I know at both suggest that there is more conspicuous wealth at Duke, possibly because the Greek system facilitates it. (Not necessarily more wealth, but this may also be a bit of a NE vs southern thing.)
Boston is a fun student city and upperclassmen in particular take advantage of it for their social life. Duke definitely has a milder climate, and that impacts how people “play” as well.
Since you have friends at both, you can ask them about everything from how people socialize, substance use, academic environment, etc. Iow, the things that may be harder to figure out from published info.