Will getting a B in math [linear algebra at a college] as a prospective math major hurt my chances at top schools?

You are over analyzing your deferral.

Hard to get into schools are hard to get into. You knew that b4 you applied.

Stop over analyzing

Ps what @blossom said.