Under FERPA, I think you are protected, and have rights and privileges which may calm your concerns:
"FERPA generally prohibits the improper disclosure of personally identifiable information derived from education records. Thus, information that an official obtained through personal knowledge or observation, or has heard orally from others, is not protected under FERPA. This remains applicable even if education records exist which contain that information, unless the official had an official role in making a determination that generated a protected education record.
Under FERPA, a school is not generally required to maintain particular education records or education records that contain specific information. Rather, a school is required to provide certain privacy protections for those education records that it does maintain. Also, unless there is an outstanding request by an eligible student to inspect and review education records, FERPA permits the school to destroy such records without notice to the student."
There is more: https://www2.ed.gov/policy/gen/guid/fpco/ferpa/students.html