Saying this only to give you hope that things can improve - my son at Worcester is enjoying his time at Oxford. I’m sure that the maths crowd is different from the PPE/E&M crowd (a lot fewer Union hacks I would bet), but he’s done a lot to broaden his relationships. He’s active with the chess team, the ski team, plays college football, basketball & tennis. They are also trying to re-boot the Oxford American Society. And yes - you should feel good about a just a few marks off distinction in your prelims. My son who was a state math champ his junior and senior year was in the same boat. My hunch is that you could hopefully improve your experience.
Please feel free to DM me if you would like to get in touch with him.