This is a very tough decision. However, I think that I agree with @Yawnmom. I went to a different but similarly very demanding university for undergrad. It took two years before it wore me down. The problem is that the work does not let up for four years.
If you want to do it, then a school such as Harvey Mudd, or MIT, or Caltech, is a wonderful experience. You can learn a lot in four years at any of these schools. You can do things that you never realized that you could do.
However, you have to want to do it. It is a huge amount of work. In my experience, there were a couple of extracurricular activities that I really wanted to do as an undergrad (sailing and fencing). My school had these. I just didn’t have time to do it. My social life suffered. Academics was life for four years.
You have figured this out more than a year faster than I figured it out. You can find schools that are a better fit for you.