Yes, Take the test. There will be no drawback. If a college accepts the score for credit then you are that much ahead.
For the teacher it can make a difference if one of the hurdles in the job description for bonuses etc. is that x percentage students in that AP class pass the class with x score.
I remember watching a couple YouTube videos (I believe she had five videos?) from a former Stanford AO who reviewed a number of applicants in each video. For each and every applicant, she writes on her white board the AP scores of the student along with their other sats, and she definitely talks about the scores as part of her review of the candidate. Maybe someone else has seen these as well?
I can’t imagine that an AO could “unsee” or disregard seeing six 5s vs a mixtures of six 3s/4s. If the college does not want AP scores to be part of their review, there would be no place to include these scores on the application. Maybe they don’t officially announce that AP scores are a significant part of their review, and I’m sure they aren’t valued more than grades and rigor, but I can easily see that they could tilt candidates toward an outcome or act as a tiebreaker between candidates.