The location means UCSC is also not at all a commuter college, or one where it’s easy to go home at weekends (unless you live close by in the Bay Area and even then public transit to San Jose is poor so you really need a car).
Although UCSC has a higher proportion of white kids than other UCs, it also has far fewer poor students, and I think the difference in perceptions is much more based on classism than racism.
Santa Cruz is very expensive and on-campus housing is limited so unless you are relatively well off, UCSC is unlikely to be affordable (my kid is paying $1500 per month including utilities for half of a double room). So the kids who go to Santa Cruz are mostly UMC and other UMC high school students see that.
Conversely the kids who go to Merced are less wealthy and often first gen, which affects choices of activities. How many are taking advantage of the proximity of Yosemite/Tahoe to go hiking or skiing at Merced vs those who take advantage of the ocean to go surfing at Santa Cruz? Add to that the higher number of commuters/students going home at weekends and the social experience can be quite different.
See this prior discussion of demographics by campus, UCSC has the highest percentage of “not low income” instate students of any UC: