I think this is a good plan.
We did some visits during “off times” and while not perfect, I wouldn’t rule that out either. My D was also in a sport that was difficult to miss. It was not school related and practices where over most weekends. While you don’t really get a sense of what campus is like when students are there you still get to see things and see the area.
Once decisions were in we took some time to visit admitted student days at her top two choices. My then her sport season was over and those days were normally very well run and gave an in depth look at the schools and their programs. And at that point she had narrowed down her list a bit. We had to miss a day or two of school for each though.