I’m not sure I agree, @EarlVanDorn. While everyone who uses the term “Jew down” is not an anti-Semite, I wouldn’t assume none are either. For too many people it means more than just seeking the best price (as non-Jews would do too), but can also imply being sly, crafty, or generally taking advantage of others.
It’s a loaded phrase. I would avoid using if, but not overreact if I heard it used.
And remember that when you’re not of a particular ethnic group, it’s generally offensive to use that ethnic group’s name as shorthand or slang for a particular behavior.