The wrong question to ask of automation in clinical informatics is whether the system is accurate. The right question is what its errors are made of. A model that is 95% accurate is reassuring until you learn that the 5% it gets wrong includes the silent, high-risk failures that reach a patient. Accuracy is an average. Safety lives in the tail.
So when we set out to test whether an agentic system could automate a clinical-informatics task, we designed the experiment around the failure that matters, not the accuracy that markets well. The task was medication reconciliation at admission: detecting discrepancies between a patient’s documented medication history and the admission record — omissions, duplications, dose mismatches, and look-alike substitutions. It is high-volume, error-prone, and consequential, which is to say representative. The experiment, dataset, and figures described here are illustrative.
The design
We built a constructed evaluation set of 2,400 reconciliation cases. Into it we planted 360 high-risk discrepancies — the dose errors, omitted anticoagulants, and confusable drug names that cause actual harm — so that the dangerous cases were present in known number and could be scored directly rather than estimated. A panel of clinical pharmacists established the reference standard.
Four design choices did the real work.
First, pre-registered success criteria. We fixed the thresholds before running anything, and we did not set them on accuracy alone. The binding criterion was a ceiling on missed high-risk discrepancies — the failure we were unwilling to tolerate — not a headline sensitivity figure.
Second, mechanical screens. Wherever a check could be made deterministic, it was, and it was placed as a hard gate the output had to pass: dose-unit plausibility, formulary existence, and known-interaction flags. Anything failing a screen was stopped and surfaced, never passed through on the model’s say-so.
Third, blinding. The clinical reviewers who rated each flag for clinical actionability did not know whether it had been produced by the system or by a human reconciler. Blinding removed the automation bias — in both directions — that contaminates most evaluations of this kind.
Fourth, adversarial cases. Beyond the planted discrepancies, we included truncated entries, free-text drug names, and deliberately confusable pairs, to probe the failure surface rather than flatter the system on clean data.
The results
On the headline metrics the system performed well: sensitivity of 0.94 and specificity of 0.91 for discrepancy detection. Had we stopped there, we would have declared victory and learned nothing useful.
The metric that governed the decision was the high-risk miss rate. With the mechanical screens active, the system missed 1.9% of planted high-risk discrepancies. With the screens disabled — the model reasoning alone — that figure rose to 6.7%. The screens were not a refinement; they were the difference between a deployable system and an unsafe one, and they worked precisely because they caught the dangerous-but-checkable cases that the model, left to itself, waved through with confidence.
The blinded panel rated system-originated flags as clinically actionable in 88% of cases, against 90% for human-originated flags — a difference within noise, and the closest thing the experiment produced to a claim of parity. For context, an unaided clinician baseline working under realistic time pressure missed high-risk discrepancies in 9% of cases, which is the standard the automation must actually be measured against, rather than an idealised clinician who does not exist on a busy admissions ward.
What the defect log revealed
The most valuable output was not a number but the defect log. Nearly every residual high-risk miss clustered in one place: non-standard free-text entries that the extraction layer mishandled. That is an actionable finding. It points to a targeted fix — improving structured capture upstream and adding a screen for unparsed free text — rather than a vague instruction to make the model better. A failure surface that is characterised is a failure surface that can be engineered against. A failure surface hidden behind an accuracy average cannot be.
A governance question, not a technical one
Read this way, the experiment is an instrument of clinical governance rather than a software benchmark. The pre-registered safety ceiling is a risk-tolerance decision that belongs to clinicians and quality leaders, not to engineers; the mechanical screens are documented controls; the blinded audit is the evidence that the control still holds; and the defect log functions as a post-deployment surveillance system, the informatics analogue of pharmacovigilance. None of these artefacts is optional once a system touches patient care, and each is a by-product of running the experiment properly rather than an additional burden bolted on afterwards.
The deployment implication
None of this licenses unattended automation, and the experiment was never meant to. What it supports is a specific, bounded deployment posture: the system as a first-pass reconciler whose flags are confirmed by a clinician; mechanical screens standing as non-negotiable gates; ongoing blinded audit rather than a one-off validation; and a defect log maintained in production, so that each new failure mode becomes a permanent correction instead of a recurring surprise.
The lesson generalises beyond medication reconciliation. Automation earns its place in clinical informatics not by being clever but by being accountable and bounded — by knowing where it fails, how often, and whether the failures are the kind that harm. The experiment is not an obstacle to deployment. It is the price of being allowed to deploy at all, and the only honest basis on which a clinician should be asked to trust the thing.