Yes, you can go to a top graduate program from a little known college. It’s what you do, not where you do it, that’s most important.
However, the where you go is a little bit important - mostly indirectly. The professors at W&M and UVa are the kinds of professors that might be friends with admissions committee members at the schools you’re applying to; they’re doing more advanced research you can join in on and learn the process; there are probably better resources and internship or summer opportunities at UVa and W&M as well.
And also, you don’t really know what you will do after college. You may want to go to grad school, but maybe senior year you feel like taking a break.
Can your family afford UVa or W&M? Normally I’m a ‘follow the money’ kind of person, but when you have two really excellent in-state options with more affordable price tags - and you only have to attend them for 2-3 years - and assuming your family can afford it, the calculus changes a bit.
Don’t worry about departmental rankings - those are usually for doctoral programs, not undergraduate programs. There’s some overlap but they are not definitely the same. All three of those universities are great in those areas.