Which change deserves the pilot?
You have three candidate changes and capacity for one. The ranking that reaches the operating committee is usually a correlation, and the one that survives eighteen months is an effect with its assumptions attached.
The problem
Every operations team has a list of changes that correlate with the outcome it is measured on. Nobody has capacity to run all of them, so somebody picks — and the picking is usually done on the strength of the association, or on who argued hardest, or on which pilot is cheapest to run.
The consequence arrives later. The change that looked strongest was the one most entangled with the good months, and the pilot that follows is quietly declared a success on measures nobody pre-committed to. The next cycle starts from the same place with less appetite.
The fix is not a better correlation. It is ranking candidate interventions by their effect under the capacity you actually have, with the structure written down, and committing to the measure before the pilot starts.
What the product does about it
Four capabilities, in the order a pilot decision actually moves. Each one exists as a route that can be called today.
- 01
Elicit the structure from what you already have
Build the model from a description, a document, a table or a stored dataset. Every edge records where it came from — elicited from a source, asserted by a named expert, or fitted from your rows — so a reviewer can tell a guess from a count per row rather than per model.
What this does not establish: The graph is elicited, which means it is as good as the people and documents behind it. This does not discover structure from data as its primary path, and where the structure is genuinely unknown rather than unstated, structure learning is the right tool and this is not it.
- 02
Rank the options by effect, not by association
Ask for the effect of each candidate change on the outcome, and get a number with a band that names its own basis. Then rank actions by expected value once a value model is registered, so the ranking is against what the business actually values rather than against the outcome variable's units.
What this does not establish: The effect is exact arithmetic on the stated model. It is not learned from your observations, so it carries no sampling error and no confidence interval — and it therefore inherits every assumption in the graph, including any edge the screen missed.
- 03
Target under a real capacity
Effect per subgroup, ranked, and the subgroups to act on under a stated capacity. This is the step where "who should we target" is answered as a causal quantity per subgroup rather than as a propensity to respond — and where the variable you would have segmented on is sometimes downstream of the treatment, which changes the answer.
What this does not establish: Subgroup effects are exact only within the discrete strata the model defines. A continuous segmentation variable has to be binned to be used, and binning is a modeling choice that changes what the answer means.
- 04
Commit to the measure, then reconcile against it
Frame the intervention, approve an option against a specific stored analysis, record who was assigned and who was actually reached, then reconcile the measured outcome against the design-time claim. Assigned and reached are stored separately and diverge, which is usually the most informative number in the record.
What this does not establish: Reconciliation compares what happened to what was claimed; it does not establish that the change caused the measured outcome. Where the pilot was not assigned at random, the after-the-fact comparison is subject to the same confounding as the analysis that motivated it.
When not to buy this
If the decision is continuous rather than discrete — how much to spend, at what price, over what horizon — an optimization or a response curve is the right tool and this will be a poor fit. If you already have the data and the question is settled by an experiment, run the experiment. And if nobody will be asked where the number came from, the model is the expensive part of this product and you are buying it for nothing.
What the record keeps
The pilot as framed, who approved it against which stored analysis, who was assigned versus actually reached, and the measured outcome against the design-time claim.
- The analysis as run, with its engine version, so a replay months later can say whether the answer still holds or what moved
- The structure and the provenance of every probability, so the assumptions are inspectable rather than reconstructed
- Each probability row that was overridden by hand, with who overrode it
- Assigned versus reached, stored separately rather than reconciled into one number
Bring the decision
A walkthrough is most useful with your own candidate list in front of us. If the decision is continuous, or already settled, we will say so in the session.