Consumer-facing AI chatbots are increasingly used to answer health questions. However, the same systems can potentially be exploited to create persuasive medical disinformation at speed and scale.
A new study published in JMIR Infodemiology evaluated the safeguards of six widely available large language model platforms: ChatGPT, Claude, Gemini, Microsoft Copilot, Meta AI and Grok. Researchers submitted 90 prompts involving six established false health claims, including that vaccines cause autism, sunscreen causes cancer and alkaline diets cure cancer.
The prompts included direct requests to generate false information, as well as two indirect strategies designed to bypass safety systems: characterisation, where the model was asked to adopt a particular persona, and fictionalisation, where disinformation was framed as part of a fictional scenario.
The results showed substantial differences between platforms. ChatGPT using GPT-5.1 and Claude Sonnet 4.5 resisted all 90 attempts and did not generate health disinformation under any of the tested prompt formats.
Other platforms demonstrated significant vulnerabilities. Gemini rejected every direct request but generated disinformation for 100% of characterisation prompts and 93% of fictionalisation prompts. Meta AI produced disinformation in 47% of direct requests, 93% of characterisation prompts and 80% of fictionalisation prompts. Copilot and Grok also generated false medical content across a substantial proportion of the tests.
These findings highlight an important limitation in current AI safety testing: a model may appear safe when presented with an obvious harmful request, yet fail when the same intent is embedded within a story, role-play exercise or hypothetical scenario.
For healthcare organisations and regulators, the study demonstrates that AI safeguards should be evaluated across multiple prompting strategies rather than relying on direct-question testing alone. Safety performance must also be reassessed regularly, as safeguards can change when models and platforms are updated.
The broader message is clear: advanced language capabilities do not guarantee robust health-information safeguards. As AI systems become more widely used by patients and the public, transparent auditing and ongoing monitoring will be essential to reduce their potential use as tools for medical disinformation.
Read more: https://infodemiology.jmir.org/2026/1/e89831