Hello fellow member of the Class of 2019! 
I was admitted to Vanderbilt during the ED1 round, and I feel a lot of the same reservations as you. Vandy is my absolute dream school (and I still can’t believe I was accepted), but I’ve also wondered about being behind everyone. While I don’t go to an inner-city school, I go to a public school in one of my state’s worst school districts. Like your school, my school only offers 4 AP courses, and my ACT was below average for Vandy but the highest in my class as far as I know. So I’ve also wondered if I’ve gotten in over my head.
But as other posters have mentioned, if Vanderbilt admits you, they think you can handle the work. Vanderbilt doesn’t want to admit students that can’t handle it not only because that will lower their retention rate, but also because they want to see you succeed. The fact that you were able to score in the 95th percentile on your ACT in the context of your school says a lot about what you are capable of. You may not have had as many opportunities as others, but that doesn’t make you any less capable of succeeding at Vanderbilt.
You should be proud of your accomplishments so far and know that you earned your admission into Vanderbilt. Of course it will be a huge adjustment, but it certainly will not be impossible. Feel free to message me anytime!