It sounds like Williams is your true first choice and given your legacy status, I would recommend that you apply ED. FWIW, my D (not an athlete nor a legacy, unhooked, from an overrepresented state) was accepted RD to Williams last year and rejected by Bowdoin, just one of example of the vagaries of holistic admissions. D had a 4.0 UW GPA, 1460 SAT (770 V / 690 M) which she submitted to Bowdoin and applied as a humanities student. We toured both schools and she did the optional Bowdoin video question and had an alumni interview.
Also, D was accepted to a competitive SLAC where she is a double legacy, whose parents have been similarly involved as your dad. Again, you can’t leap to conclusions from one school to the next, but I do think legacy status at SLACs is different than at bigger schools. Good luck!