It tries to break its own answers.
Some of the work an analyst would do to discredit an effect, done every time and reported either way. Three checks that can fail, plus a quantity that measures how much of the answer rests on the least supported part of it.
A check that cannot fail is decoration. That is the rule this battery is built on, and it is why the obvious check — a placebo treatment — is deliberately not shipped, and why a check that could not run is reported as inconclusive rather than counted as a pass.
Three checks
Each varies something that should not matter, and reports what happened when it did.
Estimator variation
varies: the estimator, holding the data fixed
If a materially different estimation strategy produces a materially different effect, the answer is resting on the method rather than on the data — and the report says so before a reviewer finds it.
What this does not establish: Agreement between estimators is agreement between methods. Two estimators can share the same blind spot, and neither knows about an unmeasured confounder.
Random common cause
varies: an independent random variable, added as a common cause
Adding a variable that cannot possibly matter should not move the estimate. If it does, the adjustment procedure is unstable and the magnitude of the movement is the useful number.
What this does not establish: The added variable is random by construction, so this tests the procedure against noise rather than against the confounding a real reviewer is worried about.
Data subset
varies: the rows, refitted on a subset
How much the estimate depends on which observations happen to be present. A large movement on a subset is a warning that the effect is driven by a minority of the data.
What this does not establish: It measures sensitivity to the subset chosen, not sampling error. There is no confidence interval behind it, because the model is elicited rather than observed.
What is not shipped, and why
A placebo-treatment check is not shipped. Against a stated model it is a tautology — replacing the treatment with a variable the model says has no effect will always show no effect, because the model supplied that answer. A check that cannot fail is decoration, and shipping it would have made this battery look more thorough while being less informative.
Three outcomes, and the middle one is not a pass
The states are deliberately distinct, because collapsing them is how a battery becomes a formality.
- passed
The check ran and the answer survived it. That is the most this word is allowed to mean.
- failed
The check ran and the answer moved in a way it should not have. A failure is a finding about the answer, not a defect in the check.
- inconclusive
The check could not run, for a reason the report names. It is reported as its own state and is never counted as a pass — otherwise every unrun check improves the result, and the battery always passes.
A quantity that judges nothing
Alongside the checks, the battery reports a fragility measure: how much the conclusion depends on the part of the model that is least supported. It is a number rather than a verdict, so it sits beside the checks rather than among them.
It carries no threshold and raises no warning of its own. What counts as fragile depends on the decision, and a system that chose a cut-off for you would be supplying a judgment it cannot see the context for.
None of this is validation
A battery that passes means the answer survived these attacks, not that the confounding assumption holds. Attacks that a model cannot detect are exactly the attacks a model cannot detect.
The confounding assumption is reported as a quantity rather than as a verdict for the same reason. Whether it holds is a judgment about the world, and this system shows you what it depends on so that someone who knows the domain can make that judgment.
So the honest summary of a passing battery is narrow: this effect was not sensitive to these three things. It is worth having, and it is not a clean bill of health.
If you have a decision to make and a number someone will hold you to, the question is not whether we can produce a number. It is whether you can say where it came from.