I am concerned about too much transferring. One option is to pursue a Master’s degree in a field that is related to your Bachelor’s, but not identical. I think that you should seriously consider finishing your Bachelor’s where you are, but taking enough classes to cover what you need to apply for a Master’s degree program in your intended field.
Regardless keep ahead in your classes, keep your grades up. This will help keep options open.
I am also wondering if you should try to talk to a professor in your most desired field in a university that teaches it. I have seen some cases (not many) where students have searched online and also asked around and found professors who were doing research in a specific area that they wanted to go into, and been able to start an email conversation. The message might be “I am very interested in what you are doing, how do I get more involved (whether at the undergraduate level, or at the Master’s level after I complete my degree)?”. The obvious hope is that the professor will end up helping you to complete whatever is needed to get to go to their university as a Master’s degree student.