Princeton and the UCs don’t consider freshman year grades. At other T20’s though, yes, you will probably have a tough sell. That doesn’t mean you shouldn’t take your shot at a couple of them, because an upward trend can be okay in a holistic review process. Just understand that you are up against competition that is going to have four consistently excellent years.
Depending on your goals, your major, and financial situation, T20’s may not even be a good fit. Look beyond the prestige and you’ll find many excellent choices.