I actually think the chances of being assaulted are greater in a bathroom labeled women only than one open to all genders. I also think that if there’s a vote and one person wants a single sex bathroom in the building it should be provided - going to another building is not an acceptable option.
What’s funny is while I didn’t mind mixed gendered bathrooms in college - which had changing areas attached to the showers - I really hate the fact that so many women bring boys (who are clearly over six years old), into the adult women’s locker room at our Y. (They are supposed to use the kid’s locker rooms.)