It did from the beginning. When the seceding states proudly wrote that they seceded because they “have had numerous and serious causes of complaint against our non-slave-holding confederate States with reference to the subject of African slavery” (Georgia) and that their “position is thoroughly identified with the institution of slavery-- the greatest material interest of the world” (Mississippi), it is hard to wash away the idea that the Confederate States (and therefore its symbols like flags) did not have anything to do with slavery or the racism around it.
https://www.civilwar.org/learn/primary-sources/declaration-causes-seceding-states