You obviously have 3 good choices, and you probably won't go wrong with any of them.
If you felt uncomfortable at Swarthmore, then don't go. You have to make this decision somehow, and that seems as good a reason as any to eliminate a school.
I think Rice definitely has the upper hand on USC in size, reputation, location, price, emphasis on undergraduates, and medical opportunities if that's something that you are into. Rice was ranked as one of the best schools for hispanic students in 2 different magazines.
Have you visited all 3 schools?