Oh, no doubt. I live near De La Salle high school in Nor Cal - historically a national football powerhouse. Some schools definitely seem to do well despite “not recruiting.” 
Still, for small, academic schools, especially but not exclusively boarding schools, they need kids who check a lot of boxes. A sports phenom who is all about one sport is less likely to find the teams up to their standards, and they won’t be able to fill out the school’s orchestra or debate team - not a good match for either the school or the athlete.
Cate for example has 150 boys total, and each of them has to play a sport each trimester. There aren’t enough pitchers to field a jv and varsity baseball team and a swim team and a tennis team and everything else. The phenom would be playing alongside a novice. For that reason, the phenom would more likely go to a day school and play club baseball.
That small school dynamic exists on the East Coast, too, of course. My guess is that Cal not having PG students contributes to a comparative de-emphasis on sports, though.