You can look up the percentage of students in fraternities and sororities in the college’s common data set section F1.
For a student not interested in them, a smaller percentage like 11%/3% at Swarthmore is less likely to be an issue than 75%/75% at Washington and Lee – although students interested in some types of activities may need to see if that activity is dominated by fraternities and sororities (e.g. at Swarthmore, a male lacrosse player not interested in fraternities may not like that most of the team is in Phi Psi).