Holding a grudge would be extremely unprofessional on the part of the department chair.
Being a teacher requires that even if you can dislike a student, you must still grade them fairly. And grading in mathematics and the sciences tends to be very objective. (The answer is either right or it’s wrong….)
Also consider that continued good performance/interest/behavior on your part could go a long way to change a professor’s initial poor impression about you.
IOW, don’t pre-judge your prof and underestimate his/her professionalism.
And if you relationship with this individual doesn’t improve–don’t ask him/her for a letter of recommendation.