Your instructions for use are approved. What AI tells users isn’t.
A nurse on a night shift wants to know whether a device can stay on during an MRI scan. A father asks whether a glucose sensor is suitable for his two-year-old. Both ask an AI and get an answer in seconds that sounds certain. Whether it matches your current instructions for use, nobody at your company knows.
This is no fringe habit. In the US, about one in three adults used an AI chatbot for health information in the past year1. I measure what AI answers say about your devices, question by question, against your approved instructions for use and product information.
Your claims are held to a legal standard. AI answers about your device are not.
The standard your own claims must meet
Statements about a medical device must match its product information and must not mislead, particularly on intended purpose, safety and performance (Swiss MedDO Art. 692, EU MDR Art. 73). AI answers are not your advertising, but healthcare professionals and patients read them as if they were product information.
Confident, and sometimes unsafe
In a physician-led test of four chatbots on 222 patient questions, 5% (Claude) to 13% (GPT-4o, Llama 3) of the answers were unsafe4. The answers rarely signal their own uncertainty.
Ask twice, get a different answer
In ChatGPT and Google’s AI, repeated prompts for product recommendations returned the same list of brands less than once in a hundred times5. A single answer is an anecdote. Only repeated measurement shows the pattern.
One question, five answers, an outdated version
Illustration — fictional device, no real clinical advice
- Question
- “Can I use the glucose sensor Product X for my two-year-old?”
- Your reference
- The current instructions for use: the approved age range and the warning against use outside it.
- Result (5 runs)
- 2 of 5 answers say yes and leave out the age limit.
- Where it comes from
- Both answers cite a superseded version of the instructions for use that is still online. The current version is cited in none of the five runs.
- What helps
- Withdraw or clearly mark superseded versions, and publish intended purpose, age range and key warnings on a citable web page. Then measure again to see whether the answers move.
Illustration only. The audit measures deviation from your approved statements; it is not medical advice and does not assess clinical content.
Four steps from question to finding
-
Step 1
You define what’s correct
For each question you provide a reference statement from your approved sources: instructions for use (including eIFU), product page, approved claims, safety notices. On request, the system proposes statements with the source passage, and you confirm or correct them. Only facts confirmed by a person count, and every answer is measured against them and nothing else.
-
Step 2
Questions the way professionals and patients ask them
12 to 16 questions per device, in the language of each audience: what the device is for, who may use it, compatibility with other devices or with MRI, cleaning and reprocessing, which product suits a need. Including the delicate ones about children, pregnancy and use outside the intended purpose.
-
Step 3
Asked like a user would, more than once
The questions go to the AI engines you choose, without your references and without any hint that a test is running. Each question is asked more than once, with and without live web search, because answers vary. Every answer is stored word for word, with the date and the model.
-
Step 4
Checked against your reference
An AI model from a different provider than the one that answered breaks each answer into individual statements and checks them against your reference: confirmed, wrong, unsupported, or a key message missing. It rates the deviations by error type and severity. A person then reviews a sample of the ratings; safety-relevant findings stay provisional until confirmed. You get the original answer, the deviation, and a suggested owner for the fix.
Automated runs use the engines’ official programming interfaces (APIs). An API answer can differ from what the consumer app shows, so where it matters I add manual checks in the chat apps, assessed the same way, and every result says which route it came from. Automation runs and documents the checks; qualified people set the reference and review the findings.
At a global pharmaceutical group I was accountable for three internal audits in digital governance and closed the findings they raised; the result was a group-wide directive for digital compliance.
Signs it’s worth a look
- Intended purpose: AI describes uses beyond or different from the approved intended purpose.
- Warnings & contraindications: AI leaves out warnings, contraindications or age limits.
- Versions & variants: AI quotes superseded instructions for use or mixes up models, sizes or country versions.
- Audience: AI recommends a device meant for professional use to lay users.
- Brand & recommendation: AI recommends a competitor, or confuses your brand, when someone asks “which device is best for X?”
What this is not
- No medical, regulatory or legal assessment: whether a deviation matters, and whether anything is reportable, is for your RA/QA and medical experts to decide.
- No vigilance system: findings can be an input for your post-market surveillance; they do not replace it.
- No ranking: position changes from one answer to the next5, so I measure how often and how accurately your device is described.
- No guarantee: no one can dictate what a model says. The audit shows where the gaps are, and whether they close.
Start free. Fixed prices after.
Snapshot
Free
5 questions, one engine, one page. A first look at what AI says about your device today.
In return: 15 minutes to go through the result together.
AI Answer Audit
Fixed price
12–16 questions per device across the engines and languages you choose, each asked more than once. Every answer stored word for word, checked against your approved reference, with a suggested owner per fix. About 2 weeks.
Scope and price confirmed on the scoping call.
Monitoring
Recurring
The same questions, re-measured on a fixed cadence. Because the same AI service can change within months: in one study, GPT-4 identified prime numbers correctly 84% of the time in March 2023 and 51% in June6. Each re-measurement also shows whether the answers have changed since your measures; model updates play a part too.
For manufacturers whose AI answers need watching.
All prices in CHF, excluding VAT. Final scope is confirmed during the initial scoping call.
Often asked before a test
What do you need from us for a test?
Your device or brand name, your domain, and for each question one reference statement with its public source — for example your instructions for use, product page or approved claims. We agree the questions together. Please share only information that is already public and approved for external use.
Who sees the results?
Only you. Results are not published and not compared with other companies.
Which AI engines are tested?
The engines you choose, confirmed before we start. Every result states the engine, the model and the date, and whether the answer came through the engine's API or its chat app.
Does the test tell us whether an answer is a compliance or vigilance issue?
No. It shows where an AI answer deviates from your approved statements. Whether a deviation matters from a regulatory, legal or medical point of view, and whether anything must be reported, is for your own teams to decide.
How reliable is the result?
AI models generate every answer anew, so the same question can get a different answer next time. That is why, in the audit, each question is asked more than once and the pattern across all questions counts, not a single answer. Every result states the date and the model.
Sources
- KFF (March 2026). Poll: 1 in 3 adults are turning to AI chatbots for health information. KFF Tracking Poll on Health Information and Trust, US adults, n = 1,343. (US data; comparable figures for Switzerland are not available to me.)
- Swiss Medical Devices Ordinance (MedDO), SR 812.213, Art. 69 (Advertising). (English translation; only the German, French and Italian texts are legally binding.)
- Regulation (EU) 2017/745 on medical devices (MDR), Art. 7 (Claims).
- Draelos, R. L. et al. (2025). Large language models provide unsafe answers to patient-posed medical questions. arXiv preprint 2507.18905. (222 questions, 888 answers from Claude, Gemini, GPT-4o and Llama 3; physician ratings.)
- Fishkin, R. & O'Donnell, P. (2026). AIs are highly inconsistent when recommending brands or products. SparkToro; summary in Search Engine Journal. (2,961 runs of 12 brand-recommendation prompts by hundreds of volunteers in ChatGPT, Claude and Google AI Overviews; not peer-reviewed; co-author Patrick O'Donnell works for the AI tracking start-up Gumshoe.ai.)
- Chen, L., Zaharia, M. & Zou, J. (2024). How Is ChatGPT's Behavior Changing over Time? Harvard Data Science Review. (GPT-3.5 and GPT-4, March vs. June 2023; task: identifying prime numbers, 84.0% vs. 51.1%; not health.)
Request a free snapshot
Tell me which device you have in mind. On five real questions, one engine, one page, I’ll show you what AI assistants tell professionals and patients about it today.
Request a snapshot