Correct. Colleges do not know where else an applicant has applied.
Many many students can not apply ED because of finances. They need to wait to see merit offers and/or want to compare offers across colleges. This is the reason many people do not support colleges offering ED simply because it gives an advantage to those who can/are willing to pay full price. I do support applying ED if one needs aid and the NPC suggests affordability.
Separately, at a school like Middlebury which does not require an essay, I expect students who add the school to their common app list at or on the RD deadline don’t do well in admissions unless they fit some criteria the school is still seeking in RD. (Colleges can see when an applicant adds a school to their common app list.) IMO the vast majority of highly rejective colleges practice yield protection.