Are you in SEAS or College? If you are in SEAS, consider an internal transfer to Columbia College or GS. The atmosphere is a lot less stressful for the non-engineering students.
I can only speak to a situation of a friend I had back in the late 1990s. I was attending a no-name city college for undergrad and working as a research assistant with the grad students. There was a grad student there who had left her program at Yale to come to this no-name city school to finish her degree. She had hated Yale. In her department, the vibe was too competitive and cut-throat. At the no-name college, her professors allowed her free reign over whatever she wanted to do, supported her fully, made her feel like a colleague instead of a lowly student. Her words. I never forgot that. Everyone else in the department assumed she was a) brilliant for getting into Yale and b) bold as heck for leaving Yale to pursue what truly made her happy. She had no regrets about the transfer and currently has a successful career in her field.
That does not speak to Yale as an institution - just her experience in one graduate school department twenty years ago. And know that all Ivies are not the same…and not all departments within the Ivies are the same – I attended an Ivy (neither Yale nor Columbia) as a grad student and the atmosphere was supportive, helpful, and not cut-throat at all.
Anyway, my point was that it is 100% okay to leave an atmosphere which is not helpful at all for your personal academic progress or mental health for a more supportive and laid-back/not stressful place. Being happy and productive is so much more important than being unhappy and therefore not nearly as productive at an Ivy (or any other school).
You will need to apply as a transfer, and I echo the comment above to keep up your grades at Columbia in the meantime.