Rigor does not mean hardest class in every subject, so don’t subscribe to this academic arms race way of thinking. What matters is that you seek rigor, enrichment, and engagement in what matters to you.
At a very rigorous, test-in type of high school, taking a couple of regular classes will not be a red flag. But this should be coupled with rigor in other areas that you are interesyed in. If you are interested in pursuing history or political science types of disciplines in college, however, you should figure out how to take it.