Actually, a program like the one at Hartford makes being on an elite team, at least in the summer, possible (or potentially, a team better than the school’s) The lake where Peddie rows has great programs too but it might be harder to participate if the school team wants to keep the best rowers.
In either case, I think you’d have enough experience on the water to be considered by a coach.
So this hasn’t made your choice for you. How about water polo???