It would be nice if he could come home to your current house for Thanksgiving his freshman year. He WILL have to make an effort to see his high school friends after that, but so long as you are willing to share him with their on holidays, it’ll happen. Social media really seems to allow kids to organize on a dime. And if you all wanted to visit the old 'hood, you can probably find an airbnb.
My college age kid increasingly meets up with friends at a vacation home, in their college city, etc., but not where he grew up. He only spent one college summer at home!
I would make a point, in your new house, of making sure his space is comfortable and his. As a college student, he will value coming “home” even if it’s not the home he grew up in.
But your reasons for moving and downsizing a bit really make sense. Hanging onto the house for a couple of weeks a year that may not even be doesn’t make sense.