Building built outside the ordinance

It is in a kind of conservation land. There are specific rules to follow to preserve the terrain in addition to the usual building codes and others. It was the condition the town imposed for not shutting down pre developed land. If the rules were followed, the garage will be about 40-50’ away from me instead of 5-7’.