A 60-year-old man who asked AI ended up in hospital after following dietary advice from ChatGPT. The man had been looking for ways to reduce chloride in his diet and replaced table salt with a chemical compound called sodium bromide. He later developed hallucinations and paranoia and required hospital treatment.
However, the chatbot conversation could not be retrieved, doctors were unable to determine exactly what advice the man had received. The case detailed in a paper, authored by researchers from Binghamton University, Stanford University, Texas A&M University and Indiana University, argues that the lack of transparency surrounding AI-generated health advice can make such cases difficult to investigate. "I think AI can be very helpful for health questions," Kaicheng Yang, one of the paper's authors, told Newsweek.
"But AI makes mistakes, and it could state wrong answers with the same confidence as correct ones. The consequences might include delaying treatment and taking the wrong medication." The man isn't alone when it comes to using AI for health questions, according to a Gallup survey of more than 5,500 U.S. adults conducted between October 27 and December 22, 2025, which found that about 14 million adults reported skipping a visit to a health-care provider after using AI. The researchers argue that current AI systems operate with little visibility into the health advice they provide, making it difficult for clinicians, researchers and regulators to identify harmful recommendations after the fact.
An OpenAI spokesperson told Newsweek: "ChatGPT is not a doctor and should never be used as a substitute for medical care, diagnosis, or treatment. "We're clear in our terms that ChatGPT isn't intended to be used for medical diagnosis or treatment. People still use it for health information, and we take seriously the need to make those outputs as reliable and safe as possible." The authors argued that harms linked to AI-generated health advice are often hidden from clinicians, researchers and regulators who would otherwise be able to identify and address them.
Yang, an assistant professor in the School of Computing at Binghamton University's Thomas J. Watson College of Engineering and Applied Science, told Newsweek that nobody knows "how dangerous" it is right now. Researchers recommend that users ask AI systems for their sources and verify information before making changes to medications, diets or treatments.
Yang told Newsweek: "We are not asking users to stop using AI for health questions. "Our advice is to treat the answer as a starting point, not a diagnosis. For example, users can leverage AI to understand terms, prepare questions for a doctor, or interpret test results." To improve accountability, the paper proposes several measures aimed at increasing transparency.
Extract — continue reading at the source.
