From your current list, Amherst, Hamilton, Haverford and Williams appear in a Princeton Review sampling, “Great Schools for Mathematics Majors.” That’s a good thing, and I think you’ll get at least one of these colleges. Amherst and Hamilton, with their notably flexible curricula, seem particularly suited to your dual interests in math and history. Based on its strength in history, a Georgetown admission would be nice as well. With respect to potential additions as you refine your list, consider Bryn Mawr.