Saying it once may be construed as a joke, but is still inappropriate. Saying it more than once is not longer “just” a joke.