OP, the question really is does it matter to WHOM? To employers, to grad schools, to your mother, to you?
I’ve worked with and met many people from Yale, Harvard, Stanford and just about any school you can name. It doesn’t matter to me but it often matters to them. They talk endlessly about their time in New Haven or the fun times they had at The Game. It matters to my uncle who went to Williams, has a daughter who went to Williams and is now a librarian (after grad school) and another who went to Bowdoin/Wesleyan/SUNY and is now a legal secretary (after grad school) and this uncle really does think he and his children are superior to any of my siblings (all public school grads or not grads). We don’t care. He cares.
If you wouldn’t feel happy unless you attend a highly ranked school, you should do it. I loved my ‘regular’ school and do not regret for a second not going to a highly ranked school. When my daughter tells people she’s going to Wyoming, people give the pity look. She loves it. One of my daughter’s friends was a very good student, won all kinds of awards for academics, theater, art, civics. She could have gone to many highly ranked schools, but she chose UCF. I was just shocked (I’m not a fan of Orlando, and UCF is a huge school), but she is thriving and even got a lead in the musical as a freshman. She wanted to stay closer to home, she got a big scholarship, she loves it.
Does it matter to employers? It might, but I really think you’ll be the same employee if you attend UCF or Vandy, as long as you are happy with your choice. If you are miserable at Vandy because it costs too much, is too far from home, doesn’t have the right vibe for you, then I think you’ll be better off in the end at UCF. If Vandy offers you the feel you want at college and you don’t care about the cost, then that’s the place for you.
There are many people who will always judge you by your school’s rank and it matters to them that you attended a #42 school rather than a #151 school. Does that matter to YOU?