The University of Chicago had a three-part fitness test including a 100-yard swim, strength elements. and cardio elements, that it eliminated in 2012 (not so long ago). In order to graduate, students either had to pass all elements of the test or take a one-quarter phys ed course addressing each area of weakness.
As of a few years ago, Washington & Lee still had a swim test.