I have visited both campus and for what it’s worth, here are some key similarities and differences I see. This is not an exhaustive list by any means.
Similarities:
Both have beautiful campus that integrate well into the surrounding urban environment. My impression is that Brown is much more compact
Undergraduate population is sized similarly (but grad population is roughly 2x as big at Vanderbilt)
Both highly selective (Vandy slightly less so) but both under 10% acceptance rate.
Both offer a great education and have a diversity of academic programs.
Nashville has an amazing music scene; the cultural amenities of Providence and Boston are also mighty fine
Differences:
Vandy offers some merit based scholarships - they are extremely competitive but they exist. Brown is need-based only.
Vandy has a significant Greek scene. Brown is phasing out Greek life.
Enthusiasm for college sports seems more vibrant at Vandy
South vs. NE: given that you are from California, you might be more comfortable on the east coast than in the south. Granted, both schools draw from all over the country. However Brown will have relatively more urban/suburban/NE students and Vandy will have a more southern-inflected student body. I think that Vandy might take you a bit more out of your comfort zone socially and culturally. That can be a good or bad thing, depending on what you seek in your college experience.
As mentioned by somebody else, Wash U at SL seems more comparable to Brown than Vandy and I’d put it on your list. WUSTL also has a sizable Jewish population if that’s important to you. I think it’s worth a closer look.
My recommendation: I’d only apply ED if I had a clear first choice that was also viable financially. Sounds like you’re not quite there yet but are leaning towards Brown.
Good luck!