Plenty to do at Haverford despite the size. My child never felt bored and had many options to choose from each weekend evening.
I’m not sure where you got the sense it is tough if you don’t play a varsity sport. Student body and campus events are very inclusive. My kid didn’t play a varsity sport but had some friends and romantic relationships with some fellow students who did. No barriers that I was made aware of. Again, very inclusive.
Very easy access to all Philly (great music, dining, and art scene) has to offer. I think it is the best of both worlds - a lovely, bucolic campus which encompasses an arboretum, easy access to shopping, many restaurants and coffee shops within walking/biking distance, and the amenities of a large city within easy reach.
As far as not being well known, it is well known and respected by those that matter - graduate programs and employers.