The method behind the numbers, and the numbers that went wrong.
Nine authored scenarios with known structure, elicited from prose by a model running locally, scored against the structure each case declares. Published with its worst case, its two denominators, and the three checks that failed.
This is a measurement of elicitation quality — whether a model reading a described scenario produces the right graph. It is not a measurement of the inference, which is arithmetic, and it is not a claim about your data.
How the run was made
One run, one machine, one model. Stated because a benchmark without its conditions is a number rather than a measurement.
- Date
- 2026-10-03
- the report is committed in the repository
- Model
- llama3.1:latest
- running locally through ollama — no hosted provider
- Prompt template
- 2026-09-1
- change the template and the run is not comparable
- Engine version
- 2
- so a result can be attributed to a version rather than to a date
- Samples per case
- 1
- no averaging across draws, so a case is one sample
- Cases
- 9
- 8 declare a structure; the ninth is a scale case
- Model calls
- 62
- 6.89 per case on average
- Tokens
- 29,941
- total across the run
- Wall clock
- 602 s
- about ten minutes, on a development machine
One sample per case is the largest single caveat on this page. A run with one draw cannot separate a case the model finds hard from a case where one draw went badly, and increasing the sample count is the obvious next measurement rather than something this run supports.
What the run produced
Structural metrics are computed over the variables each case could match, so coverage says how much of the expected network was even in scope.
- Mean edge F1
- 0.9583
- over the 8 cases with a declared structure
- Worst edge F1
- 0.6667
- one case; two of four expected edges missed
- Mean coverage
- 0.9688
- share of expected variables the model produced
- Cases completed
- 9 / 9
- 0 failures, 0 errors
- Qualitative pass rate
- 0.8812
- 22 of 25 evaluated checks
- Over all checks
- 0.8148
- 22 of 27 — two could not be evaluated
- Invariant failures
- 1
- recorded and not explained away
The two pass rates are the same run and both are correct. The reported convention divides by the checks that could be evaluated; two checks were skipped because a variable they needed did not appear in the elicited model, and dividing by all 27 gives 0.8148. Publishing only the first would be arithmetically defensible and would hide that a quarter of the gap is unevaluated rather than failed. A check that cannot run is not a check that passed.
Case by case
Per case rather than aggregated, because a mean hides the one that did not match. Eight cases declare a structure to score against; the ninth is a scale case and is listed anyway.
| Case | Edge F1 | Coverage | Note |
|---|---|---|---|
| chain-rain-slippery | 1.000 | 1.000 | an effect becomes a cause |
| collider-talent-beauty | 1.000 | 1.000 | structure right; one check still failed |
| confounder-age-stiffness | 1.000 | 1.000 | structure right; one check failed |
| fork-smoking | 1.000 | 1.000 | a classic fork |
| frontdoor-tar-cancer | 0.667 | 0.750 | the worst case, and the identification failure |
| marketing-frontdoor-awareness | 1.000 | 1.000 | a front-door case that came out clean |
| ops-downtime-churn | 1.000 | 1.000 | structure right; one check failed |
| pricing-discount-margin | 1.000 | 1.000 | the most expensive case at 8 calls |
| scale-car-engine | — | — | no declared structure; 7 variables, 6 edges, validation passed |
The three checks that failed
Named, with what each one observed. Two are the harness failing to detect an effect smaller than its own tolerance; the third is a real disagreement about identification, and it is the most useful thing on this page.
- collider-talent-beautycollider_opening
Conditioning on Fame should induce dependence between Talent and Beauty. Observed difference 0.007 against a tolerance of 0.050.
The check reports not-detected rather than wrong. The difference it looked for is seven times smaller than the threshold it uses, so the honest reading is that this check could not distinguish the collider opening from noise at this tolerance — a statement about the check's detection floor rather than about the model.
- confounder-age-stiffnessdependence
The blood-pressure effect should survive adjusting for the confounder. Observed difference 0.006 against a tolerance of 0.050.
Same shape as the first. Both failures live in the same place: a threshold of 0.05 applied to an effect an order of magnitude below it. Tightening the tolerance would not fix that either — it would need a larger difference to detect, which is a case-authoring problem rather than a code one.
- frontdoor-tar-canceridentification
Expected identified = false, because the scenario contains a shared genetic cause that is not measured. The engine returned identified = true, adjusting for nothing.
This is the real one. The engine was right about the graph it was given — in that graph nothing is unmeasured, so the effect is identified and needs no adjustment. What the run shows is that elicitation did not produce the unmeasured confounder the scenario declares, and the model then answered confidently about a world with less in it than the case assumes. It is the same mechanism the manufacturing industry page describes: a confident answer over an incomplete graph. The harness caught it because the case declares what the answer should have been.
A failing check is more informative than a passing one here, and the third is the reason this page exists. A benchmark of nine cases where everything passed would be a statement about the case authors.
What this does not measure
Four things, stated because the numbers above are quotable and the qualifications are not.
Not the correctness of inference
Edge F1 measures whether the right structure was elicited. Whether an effect computed on that structure is right is a property of arithmetic over tables, and it is tested separately and deterministically.
What this does not establish: A high F1 here says nothing about whether an answer computed from a graph is correct. That follows from the tables, and the tables are not what this run scored.
Not a claim about your content
Each case is an authored paragraph with a declared expected structure. The expectation is the author's, and the scenarios are short, unambiguous and written to be elicit-able.
What this does not establish: Nine short authored cases is not a sample of real documents. A real incident report, contract or clinical summary is longer, has its own vocabulary, and will not be scored by anything on this page.
Not a model comparison
One model, one template, one sample per case. The harness supports comparing models and templates, and this run did not.
What this does not establish: Nothing here says the chosen model was the best available, or that a different template would not do better. It says what this combination produced on this date.
Not repeatable byte for byte
Language model output varies, so the same run on the same machine can differ. The report is the artifact to compare against, not the code.
What this does not establish: A future run that is worse is not necessarily a regression in the engine, and one that is better is not necessarily an improvement. With one sample per case, neither is knowable from a single run.
The value of publishing this is not the 0.9583. It is that the identification failure is on the page, and that the two denominators are on the page, and that both are checkable in a committed file.