When someone (e.g. employer) considers school prestige, is s/he really judging you by ...

I hired many faculty members when I was department chair or on recruitment committees in a social science department. I never knew what high school any job candidate attended, and I didn’t care. I rarely knew and didn’t care what undergraduate college they had attended, but I might notice what they majored in. I cared most about the quality of the PhD program the applicant studied at, who she worked with (major professor), what skills she had (language, statistics, computers), her research and publications, her teaching experience. I didn’t care about GPA or test scores. I paid attention to letters of recommendation.

Now it could well be that applicants from the better PhD programs tended to come from more selective undergraduate institutions. We know that some colleges are feeder schools to doctoral programs. But it would do my own department no good at all to search for or prefer students who came from particular undergraduate institutions.