The only folks IME who get their undies twisted about this very issue are folks from the older set of boomers and older generationed folks, some ex-military, and/or some socially/religiously conservative southerners/midwesterners.
Last time this came up was at my Catholic elementary school where it was not only prohibited on account of politeness, but also because it wasn’t part of the school uniform.
However, there was an exception for students who were wearing hats to cover up scars from injuries, surgery, etc. Ironically, one older teacher from another class who tried giving one such student who was covered under such an exception ended up getting so loudly read the riot act by our principal that students whose classrooms happened to be on the same floor were able to hear.