You should definitely do what YOU want to do. Who knows if your son would even be coming “home” during the summers, anyway? Seems silly to hold on to a home for to avoid the possibility that one person (who won’t live there full time ever again, likely) might feel sad or wistful about it. Just my $.02.
I talk often about moving out of the United States once my youngest is through school. My kids don’t love the thought, but it’s not their decision to make.