A couple thoughts.
Smith is very very liberal and very political. If you are a middle of the road type person or even only slightly liberal you may feel uncomfortable. Don’t discount this.
If you are going into academia in math you are going to be surrounded by men. There is no getting away from this. If you go to Smith you will get four years with women in your math classes, but once you graduate and go one to graduate school you will be back into a male dominated world. As a woman who has worked in a male dominated field, it is manageable. Acting confident goes a long way. Walk in like you own the place. Many men will back down in the presence of a strong confident woman. Don’t let them talk over you or intimidate you. You are going to need to learn how to navigate this world eventually. I would not choose Smith because of this.