First, it’s only a fact to the person making the statement.
Second, It’s rude because it’s meant for the receiver to feel different, to feel awkward, to feel anything but good. It’s not a compliment. It’s meant to put the receiver on their defenses.
If I was the one receiving it, I would keep throwing questions back at the “giver of fact”, until they feel just as uncomfortable as they made me try to feel and think twice before making such ridiculous “statements of fact”. It’s something said that is trying to be a back handed compliment, but isn’t quite that even. A better thing to say is “you’re so dressed up, you look so pretty”., but even that has a “dig” quality to it.
Responses could be (said as sweet as the “fact”)
“Really? Why do you think that”
“Why, Don’t you think you’re dressed up”
“Is there a reason I shouldn’t dress up”
“What do you think I should have worn”
Or the best one would be (which I hate this saying, but in this instance is apropo): “Well, you do you”.
I would never make an “excuse” for my decision to look and feel the way I want. As I get older, I don’t put up much with stuff anymore.