Who Owns the Model's Mistakes? Closing the Governance Gap in Enterprise AI
Sooner or later, every AI system in production produces a wrong answer: a mis-extracted figure, a recommendation that misses context, a draft that states something untrue with total confidence. The model getting something wrong is expected. What turns it into a crisis is the question that follows, which is whose mistake it is. In many organizations the honest answer is nobody's, and that gap is where AI initiatives get into trouble.
The Four Questions an Auditor Will Ask
Xamun's Position Paper No. 7 puts the gap in concrete terms. Certified infrastructure, a reputable model, and a signed vendor contract can each answer where the work ran. None of them answers the four questions a regulator, auditor, or board asks next: where did this number come from, which rule applied and as of what date, who decided and why, and how much autonomy did this decision earn?
If your AI system cannot answer those from its own records, the mistake belongs to nobody, and that is the governance gap. It cannot be closed with a policy document. It has to be built into how the system works.
Put the Decision in Code, and the Draft in the Model
The first control is a fixed order of work. Code loads the data, computes every figure, applies the rules in force on the date being judged, and decides whether the run may proceed. The language model comes second, and only drafts. In the architecture Xamun describes, it is shown the names of the figures, never their values, and writes prose around placeholders. A number the model invents, or a placeholder it fails to use, fails the run. Validators then check the draft before anyone reads it.
This is the Two Minds design in practice, and it changes what a mistake can be. A model error cannot silently change a figure or a rule, because the model never held either.
Set Autonomy Per Decision, Not Per System
The second control is treating autonomy as a dial. Xamun's paper describes three stages: suggest, where the system proposes and a person acts; approve, where the system does the work and holds it for a signature; and autonomous, where it releases the result and a person audits a sample. The setting belongs to each decision, not to the system as a whole. Two rules make it safe. A blocking rule stops a run at every stage, so autonomy governs release and never the rules.
And anything irreversible or outward-facing, such as a message to a customer or a filing with a regulator, stays at approve permanently. Promotion between stages needs a record of clean runs, and a single reader marking an output wrong holds it back. That makes trust a statement about the past that can be checked, not a promise about the future.
Make Accountability a Named Role
The third control answers "who" directly. When a rule blocks a run or a draft fails a check, the case goes to a person in the role that the rule names, and the system refuses any other signatory. The person chooses, gives a reason, and the decision is recorded with a name, a role, and a timestamp. The reason stays in the record and is never passed back to the model.
This is the same logic regulated advice already uses: an accountable human is named, and AI does not change who that is. Accountability tied to a role, enforced by the system, is also what prevents the missing-owner stall before it starts.
Why the Record Matters
Each of these controls leaves a trail. A governed run records the data it loaded, every figure and how it was computed, every rule applied with its source and version, whether a model was called and what it returned, and what a person decided and why. That record is what lets you answer an auditor tomorrow.
Claims processing shows the same principle: the question is not only whether the AI was right, but whether you can show how the decision was made.
What to Ask a Vendor
Paper 7 closes with six questions for any provider of an autonomous AI system. Three travel well. Show me a number from last week's output: where did it come from? Who decided, and can I read their reason? How much autonomy does this decision have, and what did it earn it?
A vendor with a real answer will show you a record. A product that is a thin wrapper around someone else's model usually cannot, because it does not control the behavior it would be vouching for.
What to Check Before Scaling
For each AI use case in production, ask four things. Which role answers for it? What is the system allowed to decide on its own, and what is capped at approve? How is a mistake caught, and where is the correction recorded? And what would we show an auditor tomorrow? If any answer is unclear, that is the gap, and closing it is far cheaper before the first visible mistake than after.
If you're unsure who owns the outcomes of your AI systems, let's map it out together.
Comments