Will taking one lower level class jeopardize my chances of getting into Columbia (and other Ivy Leagues)?

Not all kids have the same aptitude for math. It might take longer for some to get it. The OP is not applying for STEM related major, so a slower pace of math won’t affect her/his chances that much.