IMO, yes.
in my mind, not sending looks as though you are trying to hide a score (which, let’s be honest, is what you would be trying to do.)
While Subject Tests are “recommended,” my belief is the reason many schools have switched to “recommended” is that some applicants from a lower socioeconomic status (SES) can’t afford more tests and/or the applicants attend a HS where no students apply to colleges where Subject Tests are requested, and therefore, may not have received the best advice from a GC.
If that does not apply to you, I am of the opinion that colleges at Brown’s tier are expecting Subject Tests from all other applicants. Although everyone will have a story about someone who was admitted without Subject Tests, my view in these cases is that there was probably more to the story (e.g. seriously hooked or lower SES).
FYI, there is a whole thread on the topic:
http://talk.collegeconfidential.com/college-admissions/2027962-sat-subject-tests-are-not-truly-optional-for-middle-upper-middle-class-applicants-at-elite-colleges-p1.html