Consider table 6.1 on page 153 of https://deepblue.lib.umich.edu/bitstream/handle/2027.42/109013/ajardina_1.pdf?sequence=1 . 21.8% of white people surveyed in 2010 were very worried about the changing ethnic makeup of the US, and 42.1% were somewhat worried.
A different survey (mentioned next in that paper) in 2013 of political centrists (78% white) found substantial worry and lack of hope about increasing diversity.
In this context, “white flight” at the neighborhood level should not be seen as surprising – if a white person is afraid of becoming a member of a minority group nationwide (more of an abstract concept), it is likely that s/he is more afraid of such a thing happening in his/her local area where it is actually visible in his/her daily life.