That literally happens all the time. People looked down on athletes, legacies and wealthy donors for being dumber all the time in college at least where I went, including the same people that looked down on affirmative action recruits. For example Jared Kushner.
I think affirmative action recruits actually get treated almost exactly like athletes, legacies and wealthy donors. I think that athletes, legacies and wealthy donors add more to the campus than affirmative action recruits because the affirmative action recruits are the culturally same upper middle to upper class type of person like everyone else just with a different skin color.