I’d go for the specific program assuming that (1) there’s a significant difference in quality between the two departments and (2) you’re positive you want to pursue graduate studies in that field.
Department quality can make a difference in PhD admissions. Letters of recommendation are key, and a glowing letter from someone who’s well-known in his/her field goes a long way.
That said, overall quality can make a difference as well. Many of the most prestigious schools have more funding available for student research, which is important for getting into a PhD program.