It's hard to give you advice because the schools are so different. Swarthmore is one of the most prestigious and well known LACs in the country. Hopkins is a big beast of a research university. Don't worry about name recognition--this will not be a huge detriment or benefit. If you, however, want to go to graduate school in a particular field, going to a research university helps--more well known faculty and availability of grad courses matters.
However, you can't go wrong with either, but you need to know what you want. You don't want to go to Hopkins if you want a small college with no research emphasis but great teaching. You don't want to go to Swarthmore is you're interested in theoretical research and a University feel. But this is purely a personal decision.
I should mention, however, that Hopkins is on the smaller side of research universities.