I don’t think JHS was saying that Yale had $3B in the 1970s, but rather that Yale’s endowment in the 1970s, adjusted only for inflation, would be about $3B today, rather than the ~$25B that it is now. I found a link that showed that it was actually about $1B in 1985.
Since then, Yale has hugely benefited from having possibly the greatest college endowment manager of all time, David Swensen, run the show. He has achieved an average compounded return of about 14% per year, far outpacing the S&P 500, and with considerably lower risk. He is the highest paid employee at Yale, and deservedly so.