If the fraternities’ houses are off campus, then college supervision of the fraternities is limited, since the fraternities can abandon campus recognition if the college wants them to abide by rules that they do not want to abide (as happened en masse at University of Colorado, and has happened to some chapters at some other campuses).
Private schools may have more leeway making rules in the other direction in disallowing student participation in outside organizations. For example, Princeton disallows frosh from joining fraternities and sororities, and Amherst disallows all student participation in fraternities and sororities.