AI model governance: what to document about the numbers your models produce
Governance expectations for AI and model risk differ by jurisdiction and sector, but they converge on one practical need: being able to say where a number came from and what was assumed.
5 min read
Why documentation is the common thread
Regimes such as the EU AI Act and supervisory guidance on model risk management differ in scope and in what they require. In practice, reviewers keep asking the same few questions: what does the model do, what data and assumptions does it rely on, how was it tested, and who approved its use.
Answering those questions months later is the hard part. A number that went into a decision needs a trail that survives staff changes and system upgrades.
What to record for every number a model produces
A useful minimum, for any figure that feeds a consequential decision:
- Origin: was it measured, fitted to data, elicited from an expert or a model, or assumed?
- Version: which model, template and engine version produced it?
- Question: exactly what was asked, under which assumptions?
- Status: was the question answerable from the model, and did the result survive stress tests?
- Disagreement: where sources conflicted, was that reported or averaged away?
Explainable AI starts with the inputs
Much of the explainable AI conversation concentrates on explaining a model's output after the fact. A complementary approach is to make the model's structure and each of its inputs inspectable in the first place. A causal graph is readable by a domain expert, and a source tag on every probability shows which parts are evidence and which are judgment.
Link the decision to the analysis
Documentation of the model is only half of it. The other half is a record of the decision: who approved which option, against which stored analysis, what was actually done, and how the measured result compared with what was claimed. Without it, an audit can reconstruct the model but not the reasoning.
What this does and does not give you
Good documentation makes governance reviews faster and disagreements more productive. It does not establish that a model is correct, and it does not by itself satisfy any regulation. Treat it as the evidence base that your compliance, legal and risk teams work from.
Written by the AnantCausal team. More in the guides library.
See what an auditable model looks like.
Book a walkthrough and review a model line by line, with every probability's source shown.