I guess it all depends on how you regard New Haven. Providence is only a little bit bigger and yet I find New Haven the more interesting of the two.
Btw, Middletown itself is not big, but it has a pretty well developed transit system. There’s not a big box store or chain restaurant that isn’t easily accessible - for free. Wesleyan students, however, tend to stick to downtown.