Am I crazy not wanting my son living on campus?

You’re assuming that this apartment will be oh so much more quiet than the dorm. I wouldn’t count on that. I had far worse experiences in apartments than I ever did in dorms.