Brown also has a med school and an engineering school.
I know nothing about Duke, but I can say with confidence that Brown is not an uber-competitive campus by any stretch of the imagination. It is very collaborative. And it has great facilities for neuroscience. And its students tend to be very happy.
I think you’re right, that academically you can’t go wrong either way. Do you care about location – what do you think about cold weather and snow? Do you want to live in the South, or the Northeast? How do you feel about fraternities – I think they are more prevalent at Duke than Brown.