If you have never actually taken any courses at your current college, no one will know professionally, and they only will know socially if you tell them. Your resume will only end up having the school that you graduate from and even if a transcript is requested by a future employer, you will only have two schools that you have taken courses at, if I understand this correctly.
Bottom line is don’t worry about this and should transfer closer to home if you think it would be best for you.