You have several great choices. I’ll try to address your concerns you have about Vandy. Greek Life certainly dominates the social scene at Vandy. However, the administration is trying to curb its influence (see http://talk.collegeconfidential.com/vanderbilt-university/1868189-thoughts-from-a-recent-grad-p1.html). I personally am not a fan of this decision, but others are. As far as the rich preppy Southern stereotype this really isn’t true anymore. Most of the students are on financial aid and many are from outside the South. There are rich preppy guys here, but you will also find them at the other schools on your list.
Reading your post it seems that Tufts is definitely your first choice. It is true that it isn’t on par with UPenn or Harvard, but it is still a great school in a great location. It is very hard to turn down Harvard, especially since cost isn’t a factor. If I had your choices I would definitely attend Harvard. However, if you love Tufts go there.