Swarthmore & Penn made your list. What about Haverford (&/or Bryn Mawr if you are a woman)? Those would also allow you to pop in to Penn for classes that aren’t available on campus.
What about not-Top-X institutions? A lot of what you describe about your desired environment would be found at any number of mid-sized to big-sized public Us. For your longer-range goals in philosophy, the department (&/or individual faculty members) may matter much more than the over-all name of the college/university.