I would suggest the sooner you know the better. You start taking major classes early in your Freshman year, so the longer you wait to switch, the farther behind you may be in graduating within a reasonable amount of time.
It may take a couple of quarters to meet the requirements and according to the link you need to complete your degree in 5 years.
https://eadvise.calpoly.edu/change-of-major-from-outside-engineering-to-bmed/