I hate to generalize, and I mean no offense to anyone BUT if I had to chose whether the average Ivy Leaguer could survive 4 years at West point or the average West Pointer could survive 4 years at Harvard (assuming it still only takes 4 years to graduate a civilian college--I'm old :) ) I think I'd have my money on the cadet.
It's not just about "getting in" but being able to stay in. My nephew went to Harvard and his comment was that once you got in the battle to stay in was pretty much won as long as you kept up with tuition. It took a lot for them to throw you out. At the service academies you have to prove your "metal" every day--physically and mentally. Many at the Ivies would never get accepted or be able to stay at an academy like West Point no matter how academically gifted they are. Being a "whole person" candidate/student is a lot tougher.