The type of issues the structure causes may matter with regard to enforcement, or perhaps seeking damages in a civil suit. Are there physical erosion/ flooding/ environmental damage to you? Or nuisance/noise/ traffic? Is the gym going to be a commercial business or a rental?
But, generally, I agree with others that litigation could be extremely costly or counter-productive.