Yeah, it matters very little for med school apps, or so I understand. There are hard things about it, but I don't think it's any harder than any other science.
Know this though: if your med school plans change, you need to realize this about neuroscience: you are basically majoring in art history. It is a subject that attracts abundant human curiosity and has no industrial applications. The research faculty job market for advanced degree holders is as big of a nightmare as it is for art historians. The degree itself is as useless for getting a typical "after-college" job as art history.
A neuroscience major.