I think your impression of Duke as “sport heavy” is wrong. Some people there like basketball, and Duke has a historically successful program, but Cameron Indoor Stadium is actually very small. Not that many students go to games. Far more people don’t go to a game than do go to a game. And nobody gives a flying flip about Duke Football. You can certainly watch a game at a local bar with friends if you want to, but if you don’t like sports I think you’ll be fine.
As I said on a different thread I am a UNC Alum and do not like Duke (truly, it’s the alma mater of Nixon and Stephen Miller, after all), but just wanted to correct that misimpression. The Cameron Crazies get a lot of sports coverage but they are not the majority of students at the school.