My first reaction here is NOT to be concerned about college admissions. Instead, the first thing to think about is that calculus is very useful in a significant number of other courses that you are going to want to take in the future.
The only students who should ever consider taking calculus this early are ones who are very good at math. Students who are very good at math might in the future want to study more math, or calculus-based physics, or engineering, or something else that depends upon calculus.
To me it sounds like you might not be solid on the prerequisites. How comfortable / solid do you feel about algebra, trigonometry, and pre-calculus? How did you do for example in trigonometry?