For a lower income URM, OP is correct, the prestige of the institution makes a huge difference, in particular their resources and influence/network.
The mention of wealthy classmates doesn’t limit op to being lower income (middle and upper middle would likely find a culture shock encountering wealth “for whom winter and summer are verbs”). So if op is middle class/upper middle class, the issue is not as pressing.
However wealth, and in many ways more prominent, will be present at all Ivies.
Another issue is cost: transfers get lousy aid. Even the most generous meet need colleges, transfers aren’t guaranteed the same generosity as freshmen. If you depend on a scholarship to attend Amherst and transferring depends on your getting a similar scholarship, your odds would be significantly lower.
In the meanwhile, try to make Amherst more hospitable, more like what you’d want it to be. Find clubs of interest or create one, try to get in touch with the First Gen program, invite the people you’re in class with to have lunch with you to continue the discussion…