Decide after you visit. Be sure to sit in on a couple of classes.
http://www.admissions.duke.edu/jump/...tClasses07.pdf
None of the math courses are listed as open to visitors, but I'm sure you could visit them if you let the professor know in advance.
http://www.siss.duke.edu/schedule/1190/MATH