Absolutely there's a chance that you won't be accepted. That's why there is a vetting process come Junior year. That being said, if accepted into Yale College, before you decide to matriculate I'd recommend you be in close communication (even have a portfolio review) with the arch dept.
Of course, the reason people are rejected is because their work or potential isn't up to their standards. Hope this helps. Good luck to you