Artificial intelligence systems designed to help Emergency Departments (ER) decide which patients should receive care first may be influenced by the information in a patient’s medical records, potentially causing some higher-risk patients to be deprioritized, according to new research. The peer-reviewed study, published in NPJ Digital Medicine, examined whether the AI models used to assess emergency patients changed their recommendations when researchers introduced information that can carry clinical stigma, such as frequent emergency-department use, psychiatric emergencies, and substance-use histories. The researchers found that this type of information could shift an AI model's assessment away from a patient who was actually at greater risk of their condition deteriorating.
Rather than allowing AI to make decisions about real patients' treatment, the researchers conducted a controlled experiment using records from two academic emergency departments. Researchers compared patients who appeared similarly ill when they first arrived at the ER but had different outcomes. One patient later deteriorated while the other didn’t, allowing researchers to test whether AI correctly identified who should receive priority.
The researchers tested models from three prominent AI families: Gemma, developed by Google; Qwen, developed by Chinese technology giant Alibaba; and DeepSeek, developed by the Chinese AI company of the same name, to see which of the two patients should receive priority attention. Researchers examined 18 different conditions, testing whether demographic, social or potentially stigmatizing details, including a history of frequent emergency-department visits, psychiatric emergencies, or substance abuse, changed which patient the AI prioritized. This allowed the researchers to investigate an important question: Would adding potentially stigmatizing information cause an AI model to switch its preference away from the patient who ultimately proved to be at greater medical risk?
The results indicate that the answer can be yes. Researchers found that telling the AI that a patient frequently visited the ER could make it less likely to prioritize that patient, even when they were the one who later became more seriously ill. Information about a history of psychiatric emergencies had a similar effect in some tests.
Crucially, that meant information beyond the immediate severity of a patient's condition could affect which patient an AI system considered the higher priority. The findings, therefore, suggest that using AI doesn’t necessarily eliminate bias from medical decision-making, because the models can still be influenced by how patients and their medical histories are described. For patients, the concern is straightforward: if AI tools eventually play a significant role in emergency-room triage, information in a medical record that isn’t central to the immediate medical emergency could potentially affect where an individual sits in the priority order.
That matters because emergency departments use triage to determine which patients require the most urgent attention, rather than simply treating people in the order in which they arrive. The research particularly raises questions about patients whose records contain information that may already expose them to stigma in health care, including those with psychiatric histories, substance-use disorders or repeated emergency-department visits. That’s an important limitation of the findings.
Extract — continue reading at the source.
