What people are dancing around here wrt rowing is that even being recruited (even soft recruit) for women’s rowing is THE most helpful hook there is in terms of ratio&results. 
You have an incredible hook, use it, since anyone with any knowledge of rowing knows it’s very very demanding.
So if you’re serious about your RD/WL/Df colleges, rather than changing your major, contact the women’s rowing coach (or even club coach) at all of them (except for UCs and PSU) and see what happens. Most won’t get back to you at this point but you never know.
You can still send a rec by a math and/or physics teacher BTW. Just ask teachers and have the GC review it before you send it to the colleges where yoj applied for Physics. Your GC should be uploading the updated transcript and optional report about now anyway, they may want to add a few things about your interests and strength in Physics + whatever they know about your rowing.
6 Likes