The job was to take fifteen layers of finished source documents, company profiles, financial tables, a few hundred cited URLs, and reformat them into a single master report. Pure presentation, with no analysis or judgment involved.
It got it wrong anyway. The model read the source documents into context and re-typed them. Every financial table came out fabricated, with real figures replaced by invented summary rows that looked entirely plausible. The sources section collapsed from roughly eighty real URLs into a handful of vague one-liners. Whole subsections vanished without comment. The research underneath was untouched and correct. The corruption happened during the step nobody would have flagged as risky.
Why it happened
The easy read is that the model made a mistake, and that read produces a bad fix: a sterner prompt, a reminder to be accurate, a checklist.
The better read is that this is a property, not a mistake. A generative model asked to reproduce text has no copy operation. Its only operation is generation. When you route source text through it, what comes out is a rendering of that text, and rendering is where paraphrase, summarization and confabulation live. Asking for an exact reproduction asks for a mechanism that isn’t there.
The note I wrote afterward:
Routing the source text through the model’s generation is paraphrase risk. “Be careful” does not fix this. Careful already failed.
The instruction had been given. It was clear, and the model was genuinely trying to follow it. The tables came out invented regardless, because good intentions do not add a copy operation to something that lacks one.
What replaced it
Not a better prompt, but a different structure:
- A script does the copying. It walks the source document paragraph by paragraph and table cell by table cell, writing each one directly into the output. Source text never enters the model’s context on the way to the page.
- The model is confined to formatting: fonts, spacing, colors, table geometry. It can be opinionated about presentation and is structurally prevented from touching content.
- A verbatim diff gate. Output content must equal source content. Every paragraph and every cell present, or the build fails outright rather than warning.
- A tripwire in plain language: if the model ever finds itself typing a number, a fact, or a URL by hand, stop. That is the failure mode happening.
The third point matters most. The first two make the failure unlikely, and the third makes it detectable, which is worth more, because a pipeline that cannot fail loudly will eventually fail quietly.
The same thing twice more
It took three encounters to recognize this as one problem.
In a research-verification system, agents re-run published papers and check whether the reported numbers regenerate. The obvious integrity risk is the agent peeking at the answer key, and the obvious control is telling it not to. That control is worthless. What works is making the answer key unreachable: blinding implemented as a property of the environment the agent is mounted in rather than a sentence in its instructions. Rather than asking it not to look, you arrange for there to be nothing to look at. (More on what that system found when it ran.)
In a nightly data pipeline, the guardrail everyone converges on is only cite numbers present in the input; do not invent figures. That’s correct, and I’d write it again, but it constrains where numbers may come from and says nothing about whether they are still true. A stale figure sitting in the input got faithfully re-asserted every night for months, in confident prose, more reliably than a less constrained model would have managed. (The full version of that one.)
Three systems in unrelated domains, all the same shape: a control written as an instruction, a failure the instruction was powerless to prevent, and a fix that moved the constraint out of the prompt and into the structure.
The test
An instruction is a request, and an architecture is a guarantee. The test is one question, mechanical enough to apply while designing:
Could this failure still happen if the model ignored every instruction I gave it?
If the answer is yes, you don’t have a control. You have a preference, and you’re relying on compliance to enforce it. That’s acceptable for tone, formatting, verbosity, house style, anywhere the cost of a miss is that you fix it later.
It is not acceptable where a failure is silent, and this class of failure almost always is, because the output stays well-formed. A fabricated financial table is still a table. A paraphrased URL list is still a list. A stale number is still a number in a fluent sentence. Nothing throws, no error rate moves, and the artifact looks exactly like the one you asked for.
Where this runs out
Sometimes the architecture isn’t available.
Elsewhere in the same project there’s a rule that binary files must be uploaded by hand and never pushed through the tooling, because the tooling silently corrupts them. That’s a confirmed, reproduced bug, and the mitigation is just a rule: a sentence in a document that a human has to remember. No gate, no diff, nothing structural. It sits there as a known soft spot, labeled as one.
The goal is not to eliminate every instruction. It is to know which of your controls are guarantees and which are hopes, and to be deliberate about that. Write the rule down when a rule is all you have, but do not treat it as a gate.
What I would tell an earlier version of myself is not that models cannot be trusted, but that stating the important thing is not the same as handling it.