How accurate is AI health advice? A 2026 study's answer

A 2026 Penn State study found that 76.2% of health answers from four AI chatbots were accurate, as judged by physicians. That matters because OpenAI reports that more than 230 million people a week ask ChatGPT about health and wellness. The study also tested older models than the ones most people use today, so it is a snapshot, not a verdict.

What did the Penn State study test?

The study was accepted to the 2026 ACM Conference on Fairness, Accountability and Transparency (FAccT). According to Penn State Health News, 34 Penn State participants took part in a “Diagnose-a-thon”, submitting 212 health-related prompts from patient and physician perspectives. Four large language models produced the answers: ChatGPT-4o, ChatGPT-3.5, Gemini-1.5 Pro and Llama3-8b.

Nine board-certified physicians then rated each response on a six-point scale of accuracy and potential harm. Overall, 76.2% of responses were judged accurate. Performance varied by specialty: obstetrics and gynaecology and otolaryngology fared best, while internal medicine, neurology and dermatology fared worst.

Two limits are worth stating plainly. The sample was small (34 people, 212 prompts) and drawn from one university. And several of the models tested, such as GPT-3.5 and Llama3-8b, are older or smaller than current ones, so the 76.2% should not be read as the accuracy of any specific product today.

How many people use AI for health, and what does it connect to?

OpenAI announced ChatGPT Health in January 2026, reporting that over 230 million people ask health and wellness questions on the platform each week. The dedicated space can connect to apps including Apple Health, Function and MyFitnessPal, and OpenAI says it will not use Health conversations to train its models. Its terms state the service is “not intended for use in the diagnosis or treatment of any health condition.”

TechCrunch noted a structural limitation of the technology: language models predict likely responses rather than verified ones, so they can produce confident errors. That is an argument for treating an AI answer as a starting point for a conversation with a clinician, not a conclusion.

Very large numbers of people are already asking, and an answer that is right roughly three times in four is both useful and unreliable. Context changes what a right answer is: the same question has different answers depending on a person’s age, history, medications and results.

The bottom line

• In a 2026 study, physicians judged 76.2% of 212 health answers from four AI chatbots accurate; performance varied by specialty.
• The study was small, from one university, and included older models, so it does not measure current products.
• OpenAI reports more than 230 million weekly health questions and says its Health feature is not for diagnosis or treatment.