Yes, you can decline. No college can force you to attend for any reason.
I made a thread ([link](http://talk.collegeconfidential.com/college-admissions/1914880-what-exactly-are-the-consequences-of-breaking-an-early-decision-agreement-p1.html)) last year asking about the consequences of breaking an ED agreement. From what I understand, you might have trouble getting into other colleges if it looks like you could have afforded the school and just backed out of ED because you didn’t want to go there anymore. If you’re being honest about not being able to afford it, you should be okay.