OP, I appreciate the sentiment above that "you need to figure out what part of the problem is you and what part is the college ". I can say in my case, having found a way for a not great fit school to work for me, then subsequently going to a graduate program that was a great fit, I was able to conclude after the fact that I had probably taken too much of the “blame” for my unhappiness with my undergraduate institution. Some was mine, for sure, but when I had the chance to experience what was for me a truly good fit, far less than I had burdened myself with. It had been impossible to isolate myself from the more toxic elements of the culture as they were pervasive. So do the work to figure out if you can make it work, but don’t let it get the better of you either.
I will also add the SES disparity is common but how it manifests itself can differ. It can be particularly rough at urban schools (as a generalization ) and at schools that have competitive cultures (where wealth can be the pile some choose dominate.)