As many have said, you really should be deciding based on fit.
As for the dream of being a college prof: be wary of dreams. For one, I think of the analogy with Oxbridge: many kids dream of being at Oxbridge, but Oxbridge actually wants kids who dream more about the subject they are studying than being at Oxbridge. Same deal with being a professor. It seems that you want to be a professor because of secondary aspects. But if you want to go down that route, you really should love the subject you will study. Otherwise, it just doesn’t make sense to go down that route (and there are other paths you can take to do the job that professors actually do: research and tutor/mentor/teach)*. Not to mention that a tenured position is extremely difficult to get these days. I doubt you would love the life of an adjunct.
- I am assuming you are not dreaming of sitting in endless committee meetings.