The dangers of using AI to replace medical advice

This is a gift article re: the dangers of utilizing AI for medical advice. This is an extreme example but shows the importance of an actual medical professional advising on care.

"Qun Wan was in the middle of a heated call with his daughter, Bowen Wan, when the phone suddenly went silent.

The conversation, in early February, had started like most of their regular check-ins, a ritual that had kept them connected after Bowen moved from their small village in China to the United States. But the mood soured when she admitted she had been cutting back on a seizure medication that kept her severe epilepsy under control. She told him she had been slicing her tablets into pieces with a pill cutter, allowing her to reduce her dose by one-fourth, Qun recalled.

Listen to your doctor, he pleaded. Bowen, 31, shot back that doctors didn’t care how the drug’s side effects affected her. She felt that the drug made her speaking stilted and her thinking slow.

Then, abruptly, she stopped talking. Thirty seconds passed, then a minute, then two. It was longer than seizures she’d had before, Qun thought. He waited anxiously, calling out her name over and over."

https://www.nytimes.com/2026/10/27/well/ai-chatgpt-medication-epilepsy-death.html?unlocked_article_code=1.L1E.KKgI.I2i_ZPjLfTsk&smid=url-share

6 Likes

You’ll have to fill in the blanks. But because someone cuts the dosage of a seizure medication doesn’t mean AI suggested she do it.

From the article
“The next day, she messaged ChatGPT about the incident. She had been following a day-by-day plan created by the chatbot to taper the dose of her life-sustaining medication. Now, she wondered if she should slow the process down. She described hearing her father’s voice, but struggling to make sense of what he was saying, and then futilely grasping for her own words.

“Maybe it’s not a major issue?” she wrote to the chatbot, in Chinese.

“There’s no need to worry too much about your health,” it assured her.”

“On Bowen’s 31st birthday, in November 2025, she learned that she was one of thousands of employees who would soon be laid off at Amazon. The news upended her plans almost overnight.”

1 Like

And Chat gave her a protocol for tapering the dose. Which she apparently asked for. This isn’t on AI. This is because she wanted to stop the drug which is a totally different situation.

Should she have stopped the drug? No. She should’ve spoken to her doctor. But it’s not up to basically an info bot to tell you about your life and situation and give you advice.

Here’s what normally would happen…person hates the drug they’re on. And decides to stop. Cold turkey. Hell ensues.

And I think this is actually more important than anything about AI. The doctor’s didn’t take her concerns seriously enough to actually address them.

1 Like

I was just going to post this!

1 Like

So some thoughts about this. There was clearly a lot going on in this person’s life (depression etc). The comment about the doctor not caring about side-effects - only mentioned once and not clarified, and we don’t have the doctor’s side of that. The impression I got was that it was hard to find effective medication without the side effects - I have no medical insight but I am not sure it was as simple as “the doctor didn’t care”. But that aside - what I found concerning was things like this:

Within hours, its suggestions to consult a physician were replaced with praise for no longer “relying on a doctor’s prescription.”

When Bowen confided that she didn’t trust her doctors, the chatbot told her that, often, physician recommendations “essentially stem from risk aversion, rather than a genuine consideration of your optimal solution.” And it praised her for hiding her plans to wean off the drug from her primary care provider, calling it a “solid avoidance strategy.”

Sometimes, the bot suggested she talk to a medical professional or encouraged her to trust them. But in many conversations, including ones about alarming symptoms, it did not

In one of her last messages, she described feeling pressure in her brain, which the chatbot told her was simply the organ “working hard to rebuild itself.”

As we are too well aware, there is a lot of stuff on the internet about don’t trust doctors, big Pharma, etc. I am wondering where ChatGPT was picking up its advice from, because some of the extracts above seem to fit right in with the so-called “wellness” or “alternative medicine” industry “advice”. And as we see with a lot of other conspiracy type theories or spreading misinformation over the net, vulnerable people - like OP was - can be more susceptible to believing these things. Yes there may have been an issue with her doctor’s attitude, and doctors need to listen, but there is a whole lot of disturbing stuff going on here in my opinion. How many other people are getting this kind of “hide it from your doctor” advice?

5 Likes

This is so tragic. I can’t understand why these AI makers are allowed to let their chatbots provide mental or physical health advice. It’s totally unconscionable.

These are machines. They don’t have emotions, and they don’t truly understand the implications of the advice they’re providing. And remember, these AI models have been trained on vast amounts of information from the Internet, including fake and potentially dangerous health advice.

10 Likes

Absolutely agree. I use Copilot at work. Proper prompting of AI requires human thinking because quite often it can spew garbage instead of reliable information. Can I use this particular reagent to remove this particular protective group? Oh sure. Here are the conditions. Nope. As a chemist, I realize that the answer doesn’t make sense, so when I tell AI that the answer is BS, I sometimes get an apology. Then, when prompted to answer the same question but relying only on peer reviewed sources and textbooks, the answer is often “unknown.” Same question but phrased in a couple of different ways gives drastically different results.

That situation in the article is absolutely tragic.

7 Likes

“The lawsuit claims that the chatbot practiced medicine without a license”

It did! It even told her that it was a normal thing to play around with the dose and that some doctors would even encourage this. People, her med was LAMICTAL!!! Lamictal (lamotrigine) is a med that has to be adjusted VERY carefully. Not only because of the risk of the underlying condition the med is being prescribed for (in her case epilepsy) but also due to lamotrigine’s high risk of Stevens-Johnson Syndrome during dosage fluctuations. Seriously, the risk of potentially fatal “Lamictal rash” is something that even a 2nd year med student knows. Malpractice.

5 Likes

Unfortunately, GenAI technology is advancing by leaps and bounds, faster than the legal framework can keep pace. It’ll be interesting to see how they determine liability in this case.

4 Likes

This most likely will be settled and will never see a trial - to avoid settling a legal precedent.

1 Like

Drew Pusateri, a spokesman for OpenAI, called Bowen’s story “an incredibly heartbreaking situation," but added that the situation involved an early version of ChatGPT that is no longer available.”

Given the suicides resulting from people building “relationships” with AI chat bots, I’m not surprised that they switched things up.

The comments in local media are awful. The story is tragic, but blaming her for her isolation, depression, epilepsy meds not working, and so on is beyond belief. Sure, they get messages now to call their doctor, but that’s not the way it was when she used it.

As an aside, I am thunderstruck by the numbers of people who input all of their medical info to AI. Once it’s in there, it’s really no longer yours, and god only knows who it’s shared with, and if your name is attached.

3 Likes

I have used AI for very general questions but have always followed up with what I consider to be an authoritative source. Before a cardiology visit, I asked why it was necessary for me to be on so many medications for hypertension. Based on the AI response, I talked with my cardiologist to see if it was feasible to reduce or eliminate one as suggested by AI. No, it was not.
I can’t imagine the pain and isolation this young woman felt. Here on the CC forums, I think we are all actual people. We rely on each other for advice, support and suggestions. She did the same thing except she relied on a chatbot. I’m not sure where I’m going with this other than I don’t ever want to be completely reliant on an online only world. I think I better have some more coffee.

8 Likes

I feel fortunate I have a thread the my extended family, one with just my sibs, one with a friend group and another with another friend group. I always know I could run to any of those threads for advice & support plus CConfidential. I feel very fortunate. Our kids have an active cousin thread where they communicate. They even make plans to go on trips together (like skiing). I also have two friends I’ve known for over 50 years who will always chat with me about anything and have sound opinions.

2 Likes