Yeah, I wouldn’t say Wesleyan and Hamilton are easier than Davidson. I’ve just read that Davidson has a reputation for grade deflation, but I think that was mostly for things like pre-med. I don’t think you’d have a significantly different workload at either of the 3 schools.
This mainly comes down to weather, location, class offerings, dorm & food options. Things like that. I think Colgate is an easy one to eliminate if greek life isn’t appealing to you. I also think it’s slightly less academic and more pre-professional than the others.