I don't think there is any course that causes as much trouble as physics. If you're hesitant about taking it, I wouldn't bother. Even the smartest students, if they don't have the knack for it, often do poorly in physics.
Wow - it might be hard for you so don't take it? God forbid a student should actually take a class that would make him think. *Smart* students who do poorly in physics do so because they are weak in math or not putting effort into the class.