I think an urban area would be harder for sure. I was a working class kid-not at an Ivy but at a LAC that had similar demographics. I think it definitely helped that it was a small town and almost everyone stayed on campus most of the time.
Best of luck to you.