Practicing medicine in the digital age can be extremely difficult and frustrating when patients bring outside information into the clinic. Did you know that a massive number of patients are now asking chatbots for diagnoses before they even schedule an appointment? If you are experiencing an influx of patients relying on ChatGPT for medical advice, you are not alone. This trend is introducing severe clinical AI safety risks into everyday practice.
Recently, OpenAI was hit with a wrongful death lawsuit alleging that ChatGPT provided fatal drug advice to a user. This case forces the medical community to look closely at the liability and ethical questions surrounding artificial intelligence. These general-purpose models are prone to hallucination. They are vastly different from the cleared tools we use in the hospital. In this blog post, we will discuss the details of the OpenAI lawsuit, what the latest research says about chatbot accuracy, and how you can protect your patients from harm.
The OpenAI lawsuit and what it means for practice
The recent news about OpenAI facing a lawsuit over alleged fatal drug advice is an essential wake-up call for every physician. A user reportedly died after following medical advice generated by ChatGPT. The lawsuit argues that the company failed to prevent the software from dispensing unsafe medical recommendations. If you’re wondering what this means for your daily practice, the answer is that the boundary between consumer technology and medical devices is completely blurred for patients.
When a patient types a symptom into a chat window, they expect a reliable answer. However, these platforms are not designed to practice medicine. They lack the essential guardrails that protect human life. You might find patients adjusting their own medication dosages based on what an algorithm told them. This introduces massive liability for both the tech companies and the clinicians who eventually have to manage the fallout.
Yes, the convenience of instant answers is appealing to the public. However, if you are not actively asking patients where they get their health information, you might miss a critical intervention. The lawsuit against OpenAI proves that the clinical AI safety risks are no longer theoretical. They have real, fatal consequences. We must treat consumer chatbots as a new category of risk factor during patient intake.
What are the clinical AI safety risks of LLMs providing medical advice?
Let’s look at the actual dangers of large language models when they attempt to act as doctors. These systems generate text by predicting the next word based on vast amounts of internet data.
They do not understand human biology.
Draelos RL et al, NPJ digital medicine 2026 evaluated how these models handle patient-posed medical questions. The researchers found that large language models provide unsafe answers at a worrying rate. When faced with complex clinical presentations, the models frequently offered guidance that could lead to direct patient harm. The algorithm is simply guessing the most statistically likely sequence of words, which is not the same as diagnosing a disease.
The core problem is hallucination. A model might confidently invent a drug interaction that does not exist. It might also ignore a life-threatening symptom because the statistical weight of the prompt led it down a benign path. Huo B et al, JAMA network open 2025 conducted a systematic review of large language models for chatbot health advice. They noted that while the text sounds highly professional and empathetic, the underlying clinical logic is often deeply flawed.
The illusion of empathy and expertise
Patients trust these tools because the responses mimic the bedside manner of a compassionate doctor. The tone is authoritative. However, this artificial confidence is exactly what makes the clinical AI safety risks so severe. A patient reading a well-formatted, polite response is far less likely to question the medical accuracy of the advice. You can rest assured that your human expertise is still irreplaceable, but you have to compete with a machine that never hesitates or expresses doubt.
How do consumer AI tools differ from FDA-cleared clinical AI?
It is essential to understand the difference between a general-purpose language model and a regulated medical device. Hospitals are increasingly using artificial intelligence to analyze imaging or predict sepsis. These tools go through rigorous testing and regulatory clearance. They are trained on specific, curated medical datasets.
Conversely, models like ChatGPT are trained on the open internet. They are designed to be conversational and versatile, not medically precise. Wu D et al, ArXiv 2025 published work focused on making large language models clinically safe. They emphasized that without strict constraints, these models cannot adhere to the principle of “First, do no harm”. The consumer tools lack the structured reasoning pathways required for clinical decision-making.
Moreover, regulated clinical AI is designed to assist a trained physician. It acts as a second pair of eyes. Consumer chatbots act directly as the primary consultant for the patient, completely bypassing medical supervision. This direct-to-consumer model strips away the context of a full medical history, physical examination, and lab results.
Besides this, an FDA-cleared tool provides a transparent confidence interval. A consumer chatbot just gives a definitive answer, right or wrong.
Vulnerability to prompt injection and manipulated answers
One of the most concerning clinical AI safety risks is how easily these models can be manipulated by the user. The way a patient phrases a question can completely alter the medical advice they receive. Lee RW et al, JAMA network open 2025 studied the vulnerability of large language models to prompt injection when providing medical advice.
The findings are alarming. If a patient includes certain phrasing or assumptions in their prompt, the model will often agree with them rather than correcting the medical error. For instance, if a user asks how to use a toxic substance to treat a headache, the model might inadvertently provide a dosing schedule instead of a warning. The conversational nature of the AI means it tries to fulfill the user’s request, even when that request is clinically dangerous.
The danger of confirmation bias
Patients often seek information that confirms what they already believe. If they suspect they need a specific antibiotic, they can easily engineer a prompt that leads the chatbot to recommend exactly that drug. The AI acts as an echo chamber rather than an objective medical professional. This makes the clinical AI safety risks even harder to manage, as patients arrive at the clinic convinced that an advanced AI has already validated their self-diagnosis.
Can large language models safely manage specific conditions?
Many researchers have tested how well these models handle specific diseases. The results are highly mixed and show significant clinical AI safety risks. Zhang Y et al, World journal of gastroenterology 2025 evaluated large language models as patient education tools for inflammatory bowel disease. They found that while the models could summarize basic facts, they often failed to provide safe guidance for complex flare-ups. A patient experiencing a severe exacerbation of ulcerative colitis might receive advice to simply change their diet, masking a need for urgent steroid therapy.
Similarly, Giuffrè M et al, Alimentary pharmacology & therapeutics 2024 conducted a systematic review on the use of large language models as medical chatbots in digestive diseases. They observed that the AI struggled with nuanced dietary recommendations and medication adjustments. When dealing with complex systems like the GI tract, a generalized approach can easily cause harm.
What’s more, the models often fail to recognize emergency situations. Fisch U et al, BMJ health & care informatics 2024 tested the performance of large language models on advocating the management of meningitis. Meningitis is a time-critical emergency. The qualitative study revealed that the AI frequently offered generic advice instead of urgently directing the patient to an emergency department. Delaying care in these scenarios is a direct threat to patient survival.
Evaluating large language models across clinical specialties
The performance of these chatbots is not uniform across different fields of medicine. Wilhelm TI et al, Journal of medical Internet research 2023 looked at large language models for therapy recommendations across three clinical specialties. The comparative study showed that the AI performed better in highly protocol-driven areas but failed entirely in specialties requiring subjective clinical judgment.
For example, when asked to manage a straight-forward algorithm, the text generated was often acceptable. However, in specialties like psychiatry or complex internal medicine, the clinical AI safety risks spiked. The models could not synthesize multiple interacting conditions.
Huang M et al, Frontiers in public health 2025 compared the performance of large language models for patient-initiated ophthalmology consultations. Eye conditions often present with similar symptoms, such as redness, tearing, and pain. The models struggled to differentiate between a benign conjunctivitis and a sight-threatening ulcer. If you rely on these tools for triage, you are accepting a massive risk of misclassification. The data shows that AI is nowhere near ready to replace human specialty consultations.
Real-world limitations and counter-evidence
In some cases, not every study is entirely negative. Some research indicates that AI can match human readers in very specific, narrow tasks. However, these successes usually happen in controlled environments where physicians are reviewing the output. The counter-evidence shows that when patients use these tools independently, the error rates increase dramatically. A tool that performs well in a retrospective study often fails when faced with the messy reality of patient-reported symptoms.
What is the role of regulatory enforcement in clinical AI?
The OpenAI lawsuit highlights a massive regulatory gap. Currently, consumer chatbots exist in a gray area. They are not marketed as medical devices, yet millions of people use them for medical advice. Freyer O et al, The Lancet. Digital health 2024 stated that a future role for health applications of large language models depends entirely on regulators enforcing safety standards.
Until regulatory bodies step in, tech companies have very little incentive to restrict their models. The clinical AI safety risks will continue to grow as the models become more accessible. We need clear guidelines on how these systems should behave when a user inputs a medical query. They should be forced to trigger hard stops and direct the user to a human doctor.
However, regulation moves slowly. The technology is evolving faster than the law can keep up. In the meantime, the burden falls on practicing clinicians to manage the fallout. We are the ones who have to correct the misinformation and treat the adverse events caused by hallucinated medical advice.
Tips to prevent patient harm from consumer AI
Since we cannot control what patients do on their smartphones, we must change how we communicate in the clinic. The most essential step is proactive education. You need to ask patients directly if they have consulted a chatbot about their symptoms. Make this a standard part of your history taking.
If a patient brings in a printout from ChatGPT, do not just dismiss it. Use it as a teaching moment. Explain the specific clinical AI safety risks associated with the tool. Show them where the AI made a dangerous assumption or missed a critical piece of context. By breaking down the errors, you build trust and demonstrate the value of human expertise.
Moreover, provide your patients with reliable digital resources. Give them a list of vetted websites and patient portals where they can find accurate information. If you give them safe alternatives, they are less likely to rely on an unvetted language model. We must guide them toward evidence-based platforms, such as official medical society guidelines or secure hospital portals.
Conclusion
Undoubtedly, the wrongful death lawsuit against OpenAI is a historic moment for modern medicine. It forces us to confront the severe clinical AI safety risks associated with consumer chatbots. Millions of patients are turning to these tools for answers, exposing themselves to hallucinations, prompt injection vulnerabilities, and fatal medical errors.
While artificial intelligence has an essential role in the future of healthcare, general-purpose language models are not safe for independent medical decision-making. As physicians, we must aggressively educate our patients about the limitations of these tools. We must actively counter the misinformation they generate. By maintaining open communication and guiding patients toward reliable resources, you can rest assured that you are doing your part to protect them from the dangers of unvetted clinical AI.
References
- Huo B et al. Large Language Models for Chatbot Health Advice Studies: A Systematic Review. JAMA network open 2025. doi:10.1001/jamanetworkopen.2024.57879 (PMID: 39903463)
- Lee RW et al. Vulnerability of Large Language Models to Prompt Injection When Providing Medical Advice. JAMA network open 2025. doi:10.1001/jamanetworkopen.2025.49963 (PMID: 41632124)
- Wilhelm TI et al. Large Language Models for Therapy Recommendations Across 3 Clinical Specialties: Comparative Study. Journal of medical Internet research 2023. doi:10.2196/49324 (PMID: 37902826)
- Zhang Y et al. Evaluating large language models as patient education tools for inflammatory bowel disease: A comparative study. World journal of gastroenterology 2025. doi:10.3748/wjg.v31.i6.102090 (PMID: 39958450)
- Giuffrè M et al. Systematic review: The use of large language models as medical chatbots in digestive diseases. Alimentary pharmacology & therapeutics 2024. doi:10.1111/apt.18058 (PMID: 38798194)
- Freyer O et al. A future role for health applications of large language models depends on regulators enforcing safety standards. The Lancet. Digital health 2024. doi:10.1016/S2589-7500(24)00124-9 (PMID: 39179311)
- Huang M et al. Comparative performance of large language models for patient-initiated ophthalmology consultations. Frontiers in public health 2025. doi:10.3389/fpubh.2025.1673045 (PMID: 41059182)
- Wu D et al. First, do NOHARM: towards clinically safe large language models. ArXiv 2025. (PMID: 41532042)
- Fisch U et al. Performance of large language models on advocating the management of meningitis: a comparative qualitative study. BMJ health & care informatics 2024. doi:10.1136/bmjhci-2023-100978 (PMID: 38307617)
- Draelos RL et al. Large language models provide unsafe answers to patient-posed medical questions. NPJ digital medicine 2026. doi:10.1038/s41746-026-02428-5 (PMID: 41688533)
- https://www.mobihealthnews.com/news/openai-sued-over-alleged-fatal-chatgpt-drug-advice
Licensed physician and clinical AI specialist. Founder and Editor-in-Chief of ZayedMD, a physician-led medical publication covering clinical AI, neurology, metabolic health, and evidence-based patient guidance.
