My D - a STEM major - never took a physics course in her life until this year as a sophomore. She just finished her second and final semester of it. She also took multi variable calculus this year (P/NP) as it supposedly helps with that second semester of physics. Yes, it’s been difficult - she went to a study group and maybe even an on-campus tutor on occasion. She likes the subject matter but recognized that it was going to be tough because, like you, she’d had no exposure to it. Got a B last semester - probably something similar this semester.
It’s not a mistake to take it if you need it for your major. What would be a mistake is to go into it without a serious plan for success - keeping up on the assignments, figuring out where resources are for help and plugging into them BEFORE the first big test (office hours, study groups, tutoring, etc.) and buckling down for tests.