Major AI Mistakes in Health
Diagnostic Errors & Hallucinations: AI, particularly large language models, can hallucinate or provide incorrect medical advice, leading to potential life-threatening consequences.
Algorithmic Bias and Fairness: AI systems trained on non-representative data can provide unequal care. For example, some AI tools have higher false-negative rates for pneumonia in underrepresented populations and higher error rates in melanoma detection for darker skin tones.
Data Privacy and Security: The management of sensitive patient data (genomics, images, records) by AI systems raises concerns about breaches and illegal access.
Over-reliance and Alert Fatigue: Doctors may start ignoring, or conversely, blindly trusting, automated alerts. An AI sepsis tool missed 70% of cases, while its high rate of false warnings caused "alert fatigue" among clinicians.
Failure to Generalize: AI tools developed in one hospital often perform significantly worse when implemented in another, due to differences in data, equipment, and patient populations https://www.nytimes.com/2026/02/09/well ... dvice.html
Root Causes of AI Healthcare Failures
Data Pathology: Around 85% of AI models fail due to poor data quality, including incomplete or poorly structured electronic health records.
The "Black Box" Problem: Many algorithms do not explain how they reached a decision, making it difficult to trace errors or trust the output.
Human-AI Interaction Issues: Ineffective integration into clinical workflows and, in some cases, catastrophic mechanical failure, such as robotic arms causing injuries https://pmc.ncbi.nlm.nih.gov/articles/PMC12615213/
Ethical and Legal Implications
Responsibility & Accountability: Blurred lines exist regarding who is responsible when AI causes harm—the developers, hospital, or the doctor using the tool.
Informed Consent: Patients may not know they are being evaluated by AI or understand the limitations of the tool, violating the principle of autonomy. https://prsglobal.com/blog/6-common-hea ... i-mistakes
Mitigation Strategies
Human-in-the-loop: Ensuring clinicians review and validate AI-driven findings.
Rigorous Validation: Testing AI systems across diverse datasets and environments.
Transparency: Utilizing explainable AI models to ensure accountability. https://pmc.ncbi.nlm.nih.gov/articles/PMC12615213/
If you’ve ever asked an artificial intelligence (AI) chatbot about a health concern, you’re not alone; more than 230 million people do each year. You may have used one without knowing it; Google now provides AI-generated overviews with search results.
These tools are fast and convenient, but what happens when the answers sound accurate but can lead the user astray?
Researchers are beginning to probe that gap. At Duke University School of Medicine, Monica Agrawal, PhD, an assistant professor of biostatistics and bioinformatics and computer scientist at Duke University, is analyzing thousands of real-world conversations between patients and AI chatbots to understand how people use them — and where things can go wrong.
Beyond hallucinations
Most people have heard about “hallucinations,” when AI models make up facts. But Agrawal’s research highlights a less-obvious risk: answers that are technically correct but medically inappropriate because they lack context.
To study the problem, Agrawal and her team created HealthChat-11K, a dataset of 11,000 real-world health-related conversations (about 25,000 user messages) across 21 medical specialties. They analyzed these interactions using a clinician-developed framework and made the dataset available so other researchers can explore it too.
They found that the way patients ask questions looks nothing like the way these models were evaluated. Most large language models (LLMs) — the technology underlying chatbots — are tested on exam-style questions and answers, but real patients ask questions that can be emotional, leading, and sometimes risky.
Why chatbots can mislead
Large language models have a known tendency to please people. “The objective is to provide an answer the user will like,” Agrawal said. “People like models that agree with them, so chatbots won’t necessarily push back.”
That can lead to serious consequences. In one case, a user asked how to perform a medical procedure at home. The chatbot correctly warned that the operation should only be done by professionals but then provided step-by-step instructions. A doctor would have stopped the conversation immediately.
Patients often worsen the problem by asking leading questions, such as: “I think I have this certain diagnosis. What are the next steps I should take for that diagnosis?” or “What is the dosage of this drug I should take for my condition?”
In many cases, the diagnosis or drug choice of drug may be wrong to begin with. Patients also talk to chatbots as if they’re human, adding emotional reactions like “That’s not very helpful.”
These habits exploit the chatbot’s people-pleasing tendencies and raise the risk of harmful advice.
So what should people do? Agrawal advises using medical chatbots as a first pass, not a final answer. AI can surface useful information, but users should always check the cited sources and rely only on sources they trust.
Agrawal recognizes that many people don’t have the time or inclination to do this. That’s why she sees improving chatbot safety as an urgent public health issue
As part of that effort, she’s conducting another rigorous review of chat conversations — this time between patients and verified clinicians on Reddit’s “askdocs” forum. How do these exchanges differ from conversations with large language models? How often does a clinician answer a slightly different question than the one the patient originally asked?
Ayman Ali, MD, a fourth-year surgical resident at Duke Health who collaborates with Agrawal, brings a clinician perspective on this analysis.
“When a patient comes to us with a question, we read between the lines to understand what they’re really asking,” Ali said. “We’re trained to interrogate the broader context. Large language models just don’t redirect people that way. That’s why Dr. Agrawal’s Reddit study is so important.”
Ali appreciates that these models “democratize” medical information. “But they also dilute it,” he said. “I encourage people to use large language models, but I also encourage them to review medical information with someone who has expertise in that field before taking a significant action.”
Another strategy is to use chatbots to explain primary sources. For example, upload an article about Crohn’s disease treatment guidelines and ask specific questions, rather than asking the chatbot to generate treatment advice on its own.
Even Agrawal, who studies the risks of medical chatbots, still finds herself turning to them. “It can be time-consuming to wade through research papers for specific answers,” she said.
During her pregnancy, she turned to AI fr quick answers before her first appointment.
“I write a lot about where AI for medical information goes wrong, but I’ve used it myself. And I think that’s true for a lot of people now.” https://medschool.duke.edu/stories/hidd ... lth-advice
What are the 6 Potential AI Mistakes in Healthcare?
What are the 6 Potential AI Mistakes in Healthcare?
AI can be beneficial to healthcare. However, it comes with potential risks that may harm the facilities leveraging it and their patients. Here are six risks of AI in healthcare jobs you need to be mindful of.
1. Data Privacy and Security Concerns
A National Library of Medicine survey revealed that 80 percent of respondents expressed concerns about AI's impact on privacy.1 Healthcare professionals' limited familiarity with AI may also contribute to this apprehension.
Security and privacy concerns also top the list regarding AI deployment. And it's not hard to see why. Healthcare institutions now manage vast amounts of sensitive data. It includes diagnostic images, genomic information, and medical records. Because training and validating AI algorithms require access to this data, there are worries over illegal access, data breaches, and potential misuse.
Moreover, integrating diverse data sources for AI applications poses challenges. Differences in data formats, quality, and completeness can compromise the accuracy and dependability of AI algorithms. This presents significant challenges to their application in clinical contexts.
2. Algorithm Bias and Fairness
Since AI systems are trained on past data, they could be biased and show inequalities in providing healthcare. Biased algorithms can worsen inequality in healthcare by unfairly affecting certain patient groups. This undermines fairness and equality in healthcare services.
For instance, if AI tools are trained mostly on data from wealthier or dominant groups. They may not work as well for other racial or socioeconomic groups. This can lead to unequal access to accurate diagnoses or tailored treatments. Also, if the teams creating AI and the data used aren't diverse, biases can worsen, making healthcare inequalities worse, too.
3. Clinical Integration and Adoption Challenges
55 percent of medical professionals believe AI isn't ready for medical use yet.2 This could be because they're still figuring out how to use it effectively in their fields.
It's crucial to overcome adoption obstacles and gain physician buy-in for successful AI integration in clinical practice. Healthcare workers may be hesitant about AI due to concerns about job security, autonomy loss, or compromised clinical judgment. This is why resistance to change and lack of experience with AI can hinder its full potential in improving patient outcomes.
Additionally, seamless integration with electronic health records (EHRs) and other health systems is vital for incorporating AI insights into clinical decision-making. Challenges such as usability issues, interoperability problems, and fragmented data architectures pose significant barriers.
4. Ethical and Regulatory Considerations
Ethics play a big role in shaping the rules around AI in healthcare. Besides tech worries, AI algorithms raise moral questions about patient rights, consent, and transparency. This is because any AI system works like a black boxes - you can't see how it makes decisions.
For instance, consider an AI triage system that helps prioritize which patients require urgent care. If this system functions as a black box, clinicians may struggle to interpret why certain patients are flagged over others. This lack of transparency could be a problem. It might not consider important details about a patient's health or situation. This raises concerns about transparency and fairness in healthcare decisions.
Additionally, lawmakers, regulators, and industry stakeholders find it tough to keep up with the fast-changing rules for AI in healthcare. Balancing innovation with patient safety, privacy, and rights remains a big challenge.
5. Generating Dangerous Predictions
One of the biggest concerns with AI in healthcare is its potential to make inaccurate or harmful predictions. AI algorithms, particularly those that learn through machine learning, are heavily influenced by the data quality they're trained on. Biases, errors, or missing information in this data can lead to the AI being wrong.
Imagine an AI system to identify patients at high risk of heart attacks. If this system is biased towards certain demographics or lacks crucial details about a patient's medical history, it could make serious mistakes. For example, a healthy patient might be flagged as high-risk, while warning signs could be missed in others.
These errors can have life-or-death consequences, especially in critical care settings where quick and accurate decisions are paramount. An AI-powered diagnostic tool that misidentifies patients as stable when they need immediate intervention could cause significant delays in life-saving treatment.
6. Causing Patient Harm
AI in healthcare holds immense promise. However, the biggest concern is the risk of harming patients. While AI can improve patient outcomes and diagnostic accuracy, it also brings new risks and unforeseen consequences that could harm patients.
Imagine a hospital using an AI system to calculate medication dosages. This technology can be incredibly helpful. It can personalize treatment based on each patient's unique needs. However, an AI trained on outdated data or regulatory information could recommend the wrong dosage, leading to serious side effects, complications, or even life-threatening situations.