If you argue that it costs more to pay an accountant in Boston, MA than in Lawrence, KS, that is probably true. On the other hand, the salaries for the faculty, grad students and post-docs are also generally higher in Boston. What the higher overhead rate tends to show, in my view, is that the ratio of the salary of a university accountant (non-faculty, working in grant administration) in Boston to the salary of a university accountant in Lawrence is higher than the corresponding ratio of faculty salaries.
The rates are all negotiated, so there is substantiation behind them. The days when flowers for the President’s house and the cost of sailboats/yachts could be charged to overhead accounts (cough, Stanford) are over. Still, it seems to me that the excess in the overhead percentages for the private schools vs. the public schools is a form of subsidy, in fact.
The location argument holds some weight (though see the first paragraph), but if you compare UCLA and USC, or Duke and UNC, Chapel Hill, or Princeton and Rutgers, or Boston University and U Mass, Boston, that cannot be the whole story.