Gallaudet was the first school I thought of, too. Seconding Kent State, U. of Rochester and McDaniel, too.
According to College Navigator, there are 24 four-year colleges that offer a major in American Sign Language. Here’s the list: College Navigator - Search Results
For schools where sports aren’t a major focus and that aren’t known for big party scenes, you might want to take a look at:
- Goshen (IN): About 700 undergrads
- Maryville (TN): About 1100 undergrads
- St. Catherine’s/St. Kate’s (MN): About 2400 undergrads at this predominantly women’s (95%) college and part of a consortium with other Minneapolis schools, so additional class offerings and social opportunities are available.
- Eastern Kentucky: About 12k undergrads at this residential campus
That said, there are larger schools that will have sports/party enthusiasm, but that are large enough that “your” people (whatever kind of people you’re looking for) will also be attending. A couple that you may want to look into include:
- U. of Connecticut: About 19k undergrads
- U. of Houston (TX): About 38k undergrads
- U. of Kansas: About About 19k undergrads