I think that your chances of admissions at the University of Toronto are very good with or without a reference from this professor. However, getting accepted is not worth much unless you can afford it.
Having a good reference from a U of Toronto professor who knows you well and who you have worked with would be a significant positive. The Lester B Pearson scholarship would still be a reach.