I don’t know much about investment banking or baseball, but I have been a high school teacher for close to 30 years. What I always advise students is that unless you’re at the “Yo yo Ma” level of talent in a sport/art, you always should choose your college based on your long-range career/life plans.
You say you are “committed” - did you actually commit? (In which case, it’s binding, I assume.)
But if you’re not committed, I think you should choose academics over athletics. You will still be able to play baseball in a lot of less formal ways in your new college.