What colleges are you interested in? Do they care about the IB Diploma vs. the courses? Do you get extra credits because of the diploma? For example, for SUNY Binghamton
Students in the International Baccalaureate Diploma Program may receive up to 32 credits. To receive the full 32 credits, the following conditions must be met:
The IB Diploma must be completed with a score of 30 or more points; and
The student must complete at least three Higher Level exams with a score of 5 or higher.
https://binghamton.edu/harpur/advising/transfer/ib-credits.html
Try to make CAS a priority and do it!