My understanding from your post is that you will soon have an associate’s degree from a community college, but do not currently have a bachelor’s degree. If I got this right, then where you go for a PhD is a question for well into the future. This is something that you do not need to decide now, nor anytime in the near future.
It is very common, for students who end up getting two or more degrees, for the last degree to be from a different university compared to the bachelor’s degree. For students who end up with three degrees (such as bachelor’s, master’s, and PhD) it is common for two of the three degrees to be from the same school, but it is also possible for all three degrees to be from three different schools. Just for examples, I have a sibling with a bachelor’s, master’s, and PhD, and his first two degrees were from one university, and his PhD was from a different university. I have a bachelor’s and master’s from two different schools. If I had gone on for a PhD (I very seriously considered it) then it would have been from a third school, primarily because their program was a better fit to what I wanted to do. Either approach is common. Getting a PhD from the same school where you got your bachelor’s is possible and does happen, but is quite a bit less common compared to switching universities at some point.
Also, graduate students at highly ranked universities (such as U.Penn) typically come from a huge range of undergraduate schools.
So, right now you just need to think about where you want to go to complete your bachelor’s degree.
It is also not that unusual for students to be much better students as they mature. Thus a poor record in high school and a strong record in university has occurred before. When you apply to PhD programs, then will not ask and will not care what you did in high school.
So, right now you should be thinking about where to go to complete your bachelor’s degree. How many credits carry over from your associate’s degree is an issue to be considered.
Swarthmore college is a very good school with a very strong reputation. I would expect that quite a few graduates from there go on to get some form of graduate degree. However, I think that it would most likely be a reach for you to apply right now. You might get accepted, but you should also apply to other schools where acceptance is more likely.
I am not completely familiar with your background. However, from what you have said it sounds to me like starting at community college was the exact right thing for you to do.
My suggestion is that you also consider your in-state public universities. If you are in Pennsylvania, then Penn State and Pitt might both be worth considering. If you are looking for a smaller school such as Swarthmore, then there are a very large number of very good liberal arts colleges that are a possibility. Just which ones are in your area and are good for history is something that I am not familiar with. However, the transfer advisors at your community college should be able to help you.
Perhaps the first thing that you should do, other than keep ahead in your current course load, would be to make an appointment to talk to a transfer advisor at your community college.