This was my D22’s experience as well. There were definitely upperclassman in the dorms, but a strong freshman community on her hall.
And despite what everyone says, it is very easy to move to a different dorm sophomore or junior year. My D actually lived in different dorms for each of her first three years, and is in a TA for her senior year. While many people do have house pride and stay with their dorm for three years, it is totally fine to move. Junior year, if you study abroad, they often put you in a single for the semester you are on campus. My D and her best friend requested singles near one another upon return from study abroad. (I think they could rank either being in a specific dorm, or being near a specific person.) They were issued singles next-door to each other in a dorm neither had previously lived before. They were very happy.
Housing at Vassar is part of what makes it such an interesting place. I don’t think there are two dorm rooms alike in that entire school. The rooms my daughter has had over the years were all so very different in size, shape, windows, wall/floor type, and view. In fact, when my D moved in after study abroad, we were surprised to find out that her best friend‘s single room next-door, with an identical layout, was almost twice as large as D’s room. Keeps it exciting!