But it happens in some situations.
It tends not to happen where I live, but non-white Americans are common enough around here that when people as “where are you from?”, they mean exactly that (without assuming that the other person is not American, or really wanting to ask about ethnic ancestry). However, I have encountered situations where naming a place within the US was not the expected answer when visiting a place that was about 90% white (not Latino), although (oddly) the questioner himself was non-white.
Indeed, there is a fairly well known case of a non-white guy with a not-typically-Anglo-American name who is from Hawaii, but, no matter how often he or anyone else says that he is from Hawaii, many people do not believe him.