<p>Outcome-oriented metrics are great in theory. Unfortunately, there is no Common Data Set for outcomes. The existing CDS tells us little or nothing about post-graduate outcomes. The outcome metric that I like best (despite its problems) is PhD production, for the following reasons: </p>
<ol>
<li>It measures an outcome that is directly related to the academic subjects colleges actually teach.</li>
<li>It measures an outcome that is plausibly related to the treatment effects of undergraduate education. Colleges with the highest PhD production rates are not necessarily the very most selective colleges. Colleges with the highest rates do seem to share certain features in common (such as small average class sizes and a reputation for academic rigor, which in my opinion have a plausible bearing on this outcome).</li>
<li>It measures an outcome that, in most cases, occurs within 10 years of college graduation (unlike lifetime achievements such as senior leadership positions or Nobel prizes, which presumably reflect the effects of many factors besides undergraduate academic quality)</li>
<li>It can be measured either for broad interdisciplinary areas (e.g. science and math), or in many cases for individual disciplines</li>
<li>Its data sources are not sparse (unlike Nobel or Rhodes production data, for example)</li>
<li>The NSF has collected many years of data and made it searchable on the internet at webcasar.com.</li>
<li>It can be expressed easily as a number that has clear, objective meaning (unlike, say, “faculty quality”, “student engagement”, or “administrative effectiveness” – all of which are admittedly very important)</li>
</ol>
<p>Problems with this metric:</p>
<ol>
<li>The NSF reports baccalaureate origins of PhDs in absolute numbers, without normalizing either for institution size or program size. Consumers/researchers who make these adjustments can’t cite authoritative sources and methods for their adjustments.<br></li>
<li>Although the outcomes can be attributed plausibly to treatment effects, these effects cannot easily be separated from confounding selection effects of student choices. An excellent college may have low production rates simply because many of its students choose other career paths. It may enroll many students in engineering, nursing, or other fields that do not frequently lead to doctoral degrees. Socioeconomic selection effects may be confounded with academic treatment effects.</li>
<li> The percentage of PhDs earned in strong v. weak programs cannot easily be measured and compared across colleges.</li>
<li>It isn’t an outcome that many people strongly desire for themselves or their children </li>
</ol>