Do you have offers from coaches at both schools for a supported slot in ED?
Wesleyan is much more centrally located amongst the other NESCAC schools, but I am not sure if travel distance is a significant factor for the XC and Track teams.
I agree with the posters above about visiting the campuses, meeting with members of the teams and thinking about how you like Middlebury, VT and Middletown, CT. Both are great schools and would offer excellent career and academic opportunities