I’d either seek remedy at the local building/zoning/planning dept. and failing that, speak with a RE lawyer and file a lawsuit.
I’ve known a few instances where the structure had to be torn down, just depends on the jurisdiction and other potential remedies, if any.