how will it be viewed by colleges if I go from the lowest math course in my school to ap calculus by my senior year? Will it be affected too much if the rest of my application is where I want it to be
It depends on the school, and possibly on your intended program as well.
Most applicants to the most selective colleges are going to start in honors or higher, stay at that level, and do well all four years. For engineering programs, you would want to end HS in an AP calculus course (or higher). For most colleges and/or non-engineering/CS, what you are doing is probably just fine.
If you doubled up on a math in order to get to calculus, that helps show you are stretching and challenging yourself, provided, of course, you didn’t neglect the other cores in order to do so.
The most important thing is to make every effort. For a person, nothing is impossible. You must remember that you are smarter than you believe and stronger than you look. When I had such problems, I spared no time and took additional courses on the Internet, for example, on this site https://www.coursera.org/ you can find a variety of courses. For students it is even free of charge.