I would stay with Civil Engineering because of the flexibility as nordicdad noted. You never know what will happen between now and graduation. Many of the things you would learn in a construction management curriculum can be picked up on the job. You’ll find that many professionals actually do not have degrees in construction management. The civil engineering concepts you learn will not go to waste. While you won’t be sizing steel beams, the background information you have will help you understand the engineers you work with better, and thus become a better construction manager.
Now, if you were struggling with the material, then I’d suggest otherwise, but since you’re not, a little bit of hard work won’t hurt.