Haverford is generally more is respected academically, despite the rankings. As another poster commented, there is a single level of rank that separates these two. Schools move up and down the list constantly, you need to have a historical perspective in order to really factor them in.
I personally found the video “Colby Life” on You tube very informative in conveying the attitude towards academics at Colby. It’s a documentary done by a Colby Student about student life, it is about 44 minutes long. Google “Colby Life” on You tube. (sometimes moderators remove links so its best you just google it). Haverford has a very different vibe both socially and academically.