AI Isn't Always Right: Why Customer-Facing Device Integrations Need Guardrails Before They Ship

A new audit of popular AI chatbots just delivered a number that should stop every integration team in its tracks. Nearly half of the medical advice the chatbots produced was problematic.

The study, published in BMJ Open by Nicholas B. Tiller and colleagues, tested five widely used systems: Gemini 2.0, DeepSeek V3, Meta AI (Llama 3.3), ChatGPT 3.5, and Grok 2. Each model answered 50 prompts across five subjects that attract a lot of bad information online: cancer, vaccines, stem cells, nutrition, and athletic performance. That works out to 250 responses in total.

The researchers graded 49.6% of those responses as problematic. Of that group, 30% were somewhat problematic and 19.6% were highly problematic. Performance was strongest on vaccines and cancer, and weakest on nutrition, followed by athletic performance and stem cells. Grok produced significantly more highly problematic answers than chance would predict.

If you build or maintain medical device integrations, this is not a story about chatbots being bad at trivia. It is a preview of what happens when a confident language model sits anywhere near a clinical workflow without a safety net underneath it.

The danger is not being wrong. It is being confidently wrong.

Plenty of tools make mistakes. What makes large language models different is the tone.

The audited chatbots delivered their answers with high confidence and certainty, and out of 250 prompts they refused to answer only twice. A system that almost never says “I don’t know” is a system that will hand a clinician or a patient a fabricated answer with the exact same authority it uses for a correct one.

That matters enormously in an integration context. The whole point of medical device integration is to remove friction: pull a reading off the device, push it into the EHR, surface it to the right person at the right moment. We design these pipelines to be trusted and to fade into the background.

The moment you bolt a generative model onto that flow, whether it is summarizing vitals, drafting documentation, or answering a question at a patient portal, you inherit the model’s confidence problem and place it inside a channel people already trust by default.

The study authors were blunt about it. They warned that continued deployment without public education and oversight risks amplifying misinformation. Swap “public” for “clinician” or “end user” and the warning applies directly to anyone shipping AI into a connected care setting.

Fabricated citations are a special problem in a regulated field

The audit also looked at how well the chatbots backed up their claims. The result was poor. Hallucinated and fabricated references were common enough that not a single chatbot produced a fully accurate reference list.

Think about what that means in our world. Healthcare runs on traceability. Audit logs, source attribution, and defensible evidence are not nice-to-haves, they are how integrations survive a security review and a regulator’s questions. A model that invents a citation is doing the digital equivalent of charting a measurement that never came off the device. It looks complete, it reads as authoritative, but it is fiction.

An integration that cannot prove where a piece of information came from is an integration you cannot defend. That is true for a device reading, and it is doubly true for anything an AI layer generates on top of it.

Why this lands the moment AI touches a device integration

Medical device integration is increasingly sold as an on-ramp to AI. Connect the infusion pumps, ventilators, monitors, and wearables, unify the data, and let advanced models turn that stream into insight.

The benefit is real. So is the exposure.

Integration then is the amplifier. When a pipeline works, good data reaches more people faster. When something upstream is wrong, that error reaches more people faster too. Drop an unguarded model into the middle and you have built a very efficient distribution system for confident, hard-to-spot mistakes.

There is a readability wrinkle on top of this. Every chatbot in the study wrote at a college reading level the authors rated as difficult. If the AI surface is patient-facing, the people most likely to act on a wrong answer are also the least equipped to catch it.

The guardrails and safety nets to design before you go customer-facing

None of this is an argument against using AI. It is an argument against deploying it raw. Here is what belongs in the design before anything reaches a clinician or a patient.

Keep a human in the loop for anything clinical. AI output should inform a decision, not be the decision. Make the human checkpoint a required step in the workflow, not an optional courtesy the model can route around.

Ground the model in your own data. A general-purpose chatbot answering from the open internet is exactly what the study tested, and it failed. Constrain the model to verified, integration-sourced data and approved clinical references so it has something real to stand on.

Build a refusal path and use it. Two refusals out of 250 is the failure, not the feature. Your system needs explicit conditions under which it declines, escalates, or routes to a human instead of generating an answer it cannot support.

Surface uncertainty instead of hiding it. If the model is not confident, the interface should say so plainly. Confidence scores, source links, and visible “verify this” prompts turn a black box into something a clinician can actually weigh.

Demand traceable citations or suppress the claim. If the AI cannot link a statement to a real, checkable source, it should not present that statement as fact. Treat a missing source the way you would treat a missing device timestamp.

Validate against the data the device actually produces. Test the AI layer the way you test the rest of the integration, with real edge cases, adversarial prompts, and the messy inputs your devices generate in the field.

Log everything and monitor after launch. Models drift and get retuned, and the study authors noted that future results may differ as these systems change. Comprehensive audit logging and ongoing monitoring are how you catch a regression before a patient does.

Treat the AI feature as part of your regulatory footprint. If a model is shaping clinical information, it may pull your product into software-as-a-medical-device territory. Decide that on purpose, early, with the same rigor you apply to HIPAA, cybersecurity standards, and your existing integration compliance.

The takeaway for integration teams

The headline finding bears repeating: in fields where bad information already thrives, today’s leading chatbots got it wrong about half the time, said so with confidence, almost never declined to answer, and propped up their claims with citations they made up.

Integration is what gives any of that the reach to do damage. It is also where the safeguards live. Guardrails and safety nets are not a phase two you add after launch. They are the precondition for putting AI in front of a customer at all.

Build the net first. Then let the model perform over it.

Source: “AI chatbots fail medical misinformation test, returning inaccurate and fabricated advice,” PsyPost, reporting on Tiller et al., BMJ Open.