Am I crazy not wanting my son living on campus?

OP is appeasing son in the near term, but is just enabling the perpetuation of his neuroses for the long term.