Johns Hopkins and Caltech seem similarly ranked in neuroscience - very highly, so congratulations on getting accepted to two wonderful programs.
In the sciences, some of the most important considerations are the reputation of your advisor and your lab, your program’s placement rate, and your research fit with the department. You need to work with an advisor who has a network that will facilitate post-graduation options, and you need to be able to do your work inside the department. Fit personality-wise is also important, as you want to be able to get along with others in your department professionally and benefit from the resources in the department. But location and your non-academic life are also important - you will be living here for 5+ years, and you need to make sure that you can make it for the long haul.
Ask about placement, too. Ask where the graduates of the most recent cohorts have gone. Do they do things that you want to do? Even if the program doesn’t have overall statistics on this, you can ask individual PIs about the placement of their students. If they don’t know where their students have ended up at least for the first couple years out of the program, watch out. That’s a sign of a disconnected PI. (The PI-student relationship is such a close and intense one that it’s unfathomable that a PI shouldn’t have at least a vague idea of where a student has gone within the first 3-5 years of graduation.) Look, too, at the recent assistant professor hires in neuroscience in departments in which you might like to work. Do JHU and Caltech show up? (If you want a non-academic job it’s harder to find where those people went to school, but you can ask about it in the department.)
You seem to have similar opinions of both schools but I notice differences in your rhetoric: at JHU you say that the POIs are “nice” and you feel like they will be “supportive” whereas you describe the Caltech PIs primarily as top of their field and able to teach you a lot. I’m sure the latter is true of both universities, but do you feel that the Caltech PIs will be supportive mentors? That is SO important. In the sciences (all fields, really) there are some PIs and advisors who are not great at mentoring and helping their own students achieve in science. Where do you think you would get better support to become a scientist? Is there evidence of that, in the form of successful early career scientists who are doing what you want to do from both departments?
You also mention that you prefer Caltech’s location - both the proximity to LA and the campus. Location is actually pretty important in a PhD program. My position on this has evolved considerably (particularly after doing a postdoc in a small college town, which I liked but wasn’t ideal). 5-7+ years of your life is a long time, and you want to be happy. Living near LA on a campus that is beautiful and that you love can be a great thing.
What about the current students? Did you like them better at either school? What’s the environment like - competitive or collaborative? What about library resources and physical spaces like study, computer labs with the software you need, fMRI scanners? I’m sure that both Caltech and JHU have excellent facilities in both, but it’s worth thinking about. Are there any special programs or certificates that interest you that you could do, like a concurrent MS in statistics or a certificate in diversity science or whatever? Would you prefer a smaller department or a larger one? (Personally, I would prefer a smaller one. It’s nice to know all the faculty members and most of the students in your department, and you’re less likely to get lost in the shuffle.)