Whether it is worth it depends on what you want from the experience.
Most of the students who transfer into the most selective schools in 11th grade are athletes, and they are doing it to improve their academic profile. They often are already on colllege coaches’ radar.
11th grade doesn’t have a lot of openings. It is hard for most students to understand and then meet the expectations at this kind of school. Often, students will repeat 10th to get up to speed. It is highly unlikely that you will run out of coursework at any of these schools, even with a repeat year should you choose that option. It gives you more time to experience everything a school like this offers. It is also another year of tuition!
I wouldn’t worry about LORs. Faculty tend to know students because of small class size and the residents nature of the community. These relationships are part of what you are paying for.
Why are you considering this now?