I completely understand that.
I would suggest another factor to consider - what does the school do to make their lower income kids able to access all campus resources? Do clubs cost extra money? If so do FA students have the option to ask for help with those (or do they HAVE to ask)? For example, a club that does skiing and mountain climbing and other activities that are expensive and/or require expensive gear - is there a way for lower income students to participate in that club? Does the optional orientation outdoor/camping trip cost something? Is gear provided for kids who may not have sleeping bags or tents? Is there help for book costs or bedding/dorm needs/toiletries for the lowest income kids?
Some schools do consider these things so as to allow everyone to do everything offered without feeling awkward about it. Not to eat out and go clubbing, but the campus-sponsored stuff.
Also, the whole clubbing/eating out thing is more relevant in cities I think, than schools located in more rural areas. There’s simply less to go out to in those areas.
Of course there will always be differences between kids with families of different wealth that will manifest in cars and ability to take weekend jaunts and the like, but there are factors that can mitigate the “I’m the only one who can’t afford to do anything” lonely feeling it seems you are concerned about.
PS: I wrote the above before seeing @Pizzagirl 's post. YES - all that and dorms too. Some large public Us we visited had seriously fancy dorm/apts and some seriously NOT…whereas many of the smaller(?) had everyone on campus for 4 years in a housing lottery system. Then again, the larger publics we saw had plenty of lower income kids to hang out with.