Feel for you - you are probably on a fairly challenging path ahead.
I have had the opportunity to work with a lot (usually highly gifted) teens and a shift in performance at this age is both not unusual and often difficult to assess.
So the basic scan:
It can be physical / chemical imbalance / strange kinds of thyroid issues / diet issues / gluten issues. All of these can put a kid into a brain fog and literally limit his capacities. Is he always struggling - or does it vary? If it varies, what activity / diet / etc correlates to those ups and downs?
It can be depression - from alientation, gender / sexuality issues / lack of friends / or with the highly gifted an existential angst born of thinking too much about an uncertain future. It is, unfortunately the case that he will most likely not share his issues with you. Does he have another adult in his life that you trust? He may share with you - but typically only to assuage your concern - not to share the real issue. He may not even be aware of the core issue causing the depression.
It can be learning disabilities. At this age some kids grow out of them and some are only starting to show evidence of learning issues - either because the learning issues are only now emerging - or because your son was able to compensate until the course content got more difficult.
Hard to know where to start:
First, I would find a way to discover if he is actually working 5 hours night on homework, or it just appears he is. Take him to a coffee shop so you and he can “both get some work done” and watch and see where the time goes. If he is truly working that many hours, he may be developing some processing issues / or brain fog issues and you could focus on diet, learning issues, etc.
You may want to get some testing done (sorry - will be several thousand dollars). Key point: you are NOT testing for grade equivalent learning, you want someone to test for disparities between his IQ and his academic performance. The gap tells us if there is a learning issue. The testor who says, “no problem, he is just about at grade level,” doesn’t understand. And yes, IQ tests are not always accurate, achievement tests are full of holes - but the point is to look for gaps between his intelligence and his performance.
Second, depression seems likely - regardless of physical, social, or emotional causes – when your future is looking bleak, and you are adolescent, depression is an easy slope to slide… Get him outside - literally, it often does wonders - and keep it up for a couple weeks. And keep looking for activities to engage him - try something unusual? Fencing perhaps?
If none of that helps get a professional involved - be careful with doctors suggesting anti-depressants - they often help, and can also literally make the depression much worse. Therapeutic counseling can work wonders – but expect to try out several doctors to find a person your son can trust / relate to. Sorry, this is just going to take a lot of time and a lot of false starts.
Consider an Outward Bound or NOLS program. Sometimes they need to get away from their own family to grow on their own… or they just need a seriously different change of scene.
Obviously we don’t know your family situation - how much time you have to focus on these issues, finances, other children, etc., but now is the time to get really serious about finding a solution if you can. It may take a couple years.
** And fundamentally, your son’s emotional health is WAY more important than grades, or even what year in school he is in.** Please, on behalf of families and kids I have worked with, and some who have lost kids to suicide, or just destructive cycles, re-read that sentence again. Take it to heart - and apologies from me for being pushy.
You may need to take him out of school for a year. Unschool, homeschool, travel. Get your kid back. Then worry about grades and schools and college. If he falls apart now, none of that will be successful anyway.
If he can find his way back to himself, then being a “year behind” will matter to no one.
All of this writing from me because you are describing a situation that is well beyond having a little trouble in school…
One more piece - think about to what extent you can engage your kid in this whole discussion and effort to get back on path. Some kids need to be directly involved in this work, and some need their parents to do all the kinds of stuff above, but want it to be done for them, but don’t want to talk about directly. I can’t know… but try and think through your best route with your son.
Finally, I am a huge advocate of “mom’s intuition” if you don’t feel right about suggestions from a school, a doctor, etc. trust that sense. Not blindly, but it is most often right.
Good luck - this is not an easy road ahead. May you be as tenacious as your son is worth the effort.