“Canada haven’t even consider it. Any universities suggestions that would be a step up from our NJ options?”
I just took a quick look at the top schools in Canada for mathematics (I was a math major, and originally came from Canada). One ranking is here:
https://www.macleans.ca/education/best-mathematics-universities-in-canada-2019-ranking/
Of these, the top ones (UBC, Toronto, Waterloo, McGill, McMaster) are going to be significantly more than $30,000 per year for an international student even in US dollars (I am assuming you are not a citizen of Canada or France). I was a bit surprised by how much the cost has gone up for international students at McGill. Alberta is far away, quite cold, and probably just over $30,000 per year. The U of Montreal teaches classes in French. This takes you down to Victoria and SFU (8 and 9 on the list). These are very good schools and probably close to Rutgers in both academic quality and cost for you. However, they are a long way away from New Jersey (both are in British Columbia).
Rutgers is a very good university. This makes it hard to beat (assuming your son can get accepted). I don’t think that there are any in Canada that are going to match it on price while beating it on academics. There are some (such as Victoria and SFU) that will be close on both counts.