Having a physics background is an excellent way to learn how to build models of physical (and economic) systems. However, the bottom line is that you should choose something that is interesting to you and if physics fits the bill (it did for me), then go for it.