Get the emails of a couple of psych professors and go to the weekly department open house-you will know quickly where you fit in. Keep in mind that your graduate school in psych will likely not be Harvard if you decide to go there. Duke takes their own, but Harvard usually doesn't so check that out if it is important to you.
By the way,I love Davidson. We have Harvard, Duke, and Davidson in the immediate family, and the best education of the three is Davidson because of the degree of accessibility of the faculty. Duke and Harvard are both quite large, and so it is possible but much harder to spend serious time with a prof if you are not at the grad level. In fact, Harvard still hasn't completely figured out how to make the undergrad experience more personable. When I spent time there, the biggest problem was avoiding t/a's for classes. I've heard it's better now but not perfect. Duke is much better in terms of contact, and Davidson is without peer in this regard.
As a negative, it took me two years at Harvard to realize that everyone on campus was not better than me- and socially that was a very lonely time. I will say that athletics at Duke are incredible if you care about school spirit-it's a unique combination of East Coast Academics and Big 10 athletics. Anyway, spend time with your chosen department and you will know your answer.