Given your situation, I guess the courseload makes sense - I don’t think downgrading your CS class makes sense - I would move it a semester later before I downgraded.
What is currently slated for your Spring schedule? Is it a lot more math classes or is there more space?