Whether should be a factor. Everything you care about should be a factor. It’s your life and you are the one who has to live.
I wouldn’t let weather override academic quality or affordability, but all other things being equal, I think weather is a perfectly legitimate concern. I personally couldn’t live in Arizona because I wilt in the heat. 