We put the model on the happy path. Then a number appeared that wasn't real.

The first version was a show-off. You dropped a messy visit note into a box. Two seconds later you got a clean summary: complaints, meds, follow-ups, the kind of paragraph a tired person wants to paste.

We were not putting this in front of patients. It was a prototype on a patient-portal build, the sort of thing you wire up because the demo calendar is next Thursday and everyone has watched the same keynote. I keep telling clients that. I still felt the little rush when the summary looked right.

Then someone on review highlighted a creatinine value.

It was specific. It had a unit. It sat in the summary like it had always been there. It was not in the note. It was not in the structured fields. The model had been helpful in the way a confident intern is helpful: it completed the picture.

I closed the laptop and sat there for a minute like I had left a stove on.

The mistake was not the model

I do not think Gemini, or Claude, or whoever we swap in next month, is uniquely reckless. The mistake was architectural. We put generated text on the happy path. The UI treated the summary as the thing you save. There was a spinner, then a result, then a primary button that said Save.

That button is the whole story. If the primary action persists model output, you have already decided the model is a source of truth. Everything after that is damage control.

We changed the button first. It now says Insert draft. The saved record stays human. The model gets a side panel, a warning, and a list of spans it thinks it used.

type DraftSummary = {
  text: string;
  citations: Array<{ quote: string; field: string }>;
  warnings: string[];
};

function toDraft(note: VisitNote, modelText: string): DraftSummary {
  const citations = extractCitedSpans(note, modelText);
  const warnings = [];

  if (hasNakedNumber(modelText) && !numberExistsIn(note, modelText)) {
    warnings.push("A number in this draft is not in the source note.");
  }

  return { text: modelText, citations, warnings };
}

Is hasNakedNumber elegant? No. It caught the next two incidents in the same week. I will take an ugly guard over a pretty summary.

What we refuse to let the model do

I wrote this on a sticky note and later put it in the repo, which is how most of our standards get born.

• The model does not write the record. A person does.
• If it cannot point at a span, the claim does not ship.
• If the request takes too long, we show the original note. We do not show a half-summary and hope.
• Structured fields beat prose. If the value lives in a form, we read the form. We do not ask the model to remember it.
• Logs keep the prompt, the output, and who accepted it. "The AI did it" is not an audit trail.

The demo problem

Demos punish this kind of honesty. A draft with yellow warnings looks less magical than a confident paragraph. I have had a room go quiet when I showed the cautious version. Then I showed the invented number from the first build. The room got louder in a better way.

If you are selling AI as a feature, you will be tempted to hide the seams. Hide the seams in the animation. Do not hide them in the data.

We also had to talk about money, which nobody wants to do while the prototype is still cute. A summary helper that runs on every screen open is a burn rate. A helper that runs when someone asks, with a cache keyed on the note hash, is a product. Same model. Different bill.

How I explain it to non-engineers now

I do not say "hallucination." I say the model is a junior who has read a lot and will still invent a detail to make the sentence land. You would not let that junior sign the chart. You might let them draft the email.

That sentence has saved me more arguments than any architecture diagram.

We still use models. I use them every day on this site, in Flutter boilerplate, in the boring middle of a PR. I like them. I just will not put them on the path where a wrong number looks like a fact.

If your AI feature has a green primary button that saves generated text, you are not shipping intelligence. You are shipping a first draft with good posture. Change the button. The rest of the design gets easier after that.