Of course that comes back to the department to list the prerequisites for this course properly (and enforce them if there are too many students who register for the course without the prerequisites out of wishful thinking and then fail because of that). Your school cannot be any less selective than open admission community colleges; the latter do list math prerequisites for courses similar to the one you teach, in order to avoid students registering for the course and then being essentially guaranteed to fail due to insufficient math knowledge and skill.
If you counted only students who had sufficient prerequisite math knowledge for your course, would the D/F rate be much lower?