The downside is the loss of the money you’ve invested in this semester-- both in terms of tuition and room/board.
But if you honestly think it’s effecting your health, then it’s a no brainer… go home.
Why not call mom and dad tonight to talk it over? They may be able to give you some perspective that strangers can’t.