Rumination like this can lead to depression and can be a sign of anxiety and OCD. And I say that as a parent of kids with clinically diagnosed anxiety disorders. Do not hang onto this and do not feed it. You can’t go back in time and he can’t either. Tons of kids are super successful and happy and content not being red-shirted or even skipping grades or graduating on a different schedule from their peers.
If you want to help your son, suggest therapy and look into it for yourself.