Ontario tested 20 AI scribes. All 20 produced inaccurate notes.

When Ontario’s government went shopping for AI scribes, it did something unusual: it tested them. Evaluators ran simulated patient recordings through the systems of every vendor approved for procurement and read the notes that came back.

The results are in the Auditor General of Ontario’s 2026 special report on the government’s use of artificial intelligence, and they deserve more attention than they’ve received. All 20 of the approved vendors’ systems produced notes with at least one type of inaccuracy.

The three failure patterns

The evaluators’ findings grouped into three themes:

• Incorrect information. Notes generated by 12 of the 20 systems (60%) captured a different drug than the one the doctor prescribed in the recording.
• Hallucinations. 9 of 20 systems (45%) fabricated information — referring the patient for therapy, ordering blood tests, or noting “no masses found” when none of it was said in the recording.
• Missing information. 17 of 20 systems (85%) missed key details about the patient’s mental health issues in at least one of the two test scenarios, even though those issues were discussed in the recording.

These are not cosmetic errors. A different drug in the chart, an examination finding that never happened, a mental-health disclosure that vanished — each is the kind of mistake that changes downstream care.

The oversight gap

Two of the report’s quieter findings matter as much as the error rates:

• Doctors were not required to attest that they had verified a system-generated note through a sign-off feature before it entered the record.
• 11 of the 20 approved vendors did not submit any third-party audit reports of their systems.

In other words: systems known to produce inaccurate notes, with no enforced human review and, for most vendors, no independent audit trail. The report notes that the UK’s NHS England has issued guidance to minimize exactly these inaccuracies; Ontario’s procurement had no equivalent guardrail at the time of testing.

The context: the tools are needed — that’s the problem

None of this is an argument against AI drafting. The burden it addresses is real and measured. Canada Health Infoway’s evaluation of its national AI Scribe Program — over 10,500 primary care providers enrolled as of September 2025 — found that 89% of respondents reported symptoms of stress or burnout, and that providers spend about 10 hours a week on administrative work during clinic hours plus 7.2 more after hours.

But that same evaluation shows who these tools were built for: 85.6% of the program’s participants are family physicians, and the supporting evidence base comes largely from routine, single-problem visits. The dominant AI-scribe design — ambient audio in, visit note out — is a primary-care design, tested on primary-care encounters.

What this means for specialists

A specialist’s core document isn’t a visit transcript. It’s a consult letter: a referral package in, multi-system reasoning in the middle, and a letter back to the referring physician at the end. Errors in that letter don’t stay in one chart — they propagate to the colleague who acts on it.

The Auditor General’s findings frame the requirement precisely: if a drafting system can silently swap a drug or invent a finding, then the physician’s review is the safety mechanism — and review is only fast enough to be real when the system shows why it wrote what it wrote. A draft you can’t verify quickly doesn’t remove the documentation burden; it moves it from writing to policing.

That is the design constraint Consultologist is built around: every section of a consult draft generated against explicit writing standards, with the reasoning in view, and nothing entering the record before the physician signs off.

Sources

• Office of the Auditor General of Ontario, Use of Artificial Intelligence in the Ontario Government, Special Report (2026) — auditor.on.ca
• Canada Health Infoway & Women’s College Hospital Institute for Health System Solutions and Virtual Care, Evaluating AI Scribes in Canadian Primary Care (2025) — infoway-inforoute.ca