@calmlywaiting, I wouldn’t sweat it. It depends more on which is the better college for YOU.
For the record, after college visits, my daughter chose not to apply to Harvard but did apply to Swarthmore. The college and the programs of interest suited her better.
Good luck with your search and application process!