In fairness, I’m not sure anybody at Yale is really saying that this use of the term “Master” is racist, or really that it’s offensive in the way a slur is. I think the argument is that black kids in particular feel uncomfortable calling another person “Master.” It is weird–I can remember a black kid in my year who joked about it–hey, that’s my Master over there. But to some it may not seem that funny. I still think that somebody with the smarts and drive to get admitted to Yale should be able to get past this, but I don’t find it to be totally ridiculous.
There’s a much more interesting discussion at Yale, over whether to rename a residential college named after John C. Calhoun. That one has some pretty good arguments on both sides.