Contact Discuss an opportunity
Proof Engine cover card reading 'The graph moved. What caused it?' over two black rounded panels split by a glowing red seam

A Pilot Review Needs A Counterfactual (Founder's View)

As a founder, I want a positive pilot result to mean that the product worked.

The team completed the workflow faster. Follow-up improved. Users accepted the output. A manager says the process feels more controlled. These are encouraging outcomes, and after weeks of implementation it is natural to read them as proof.

But a positive result and a product-caused result are not the same thing.

The team may have paid more attention because the pilot was new. A manager may have intervened more often. The workload may have changed. A new employee may have joined. The baseline week may have been unusually poor. The process redesign around the product may have created the improvement even if the software itself contributed less than expected.

A useful pilot review needs a counterfactual: what would probably have happened without this intervention?

This does not require every founder to run a randomized controlled trial. It requires enough comparison discipline to avoid turning hope into a claim.

Founders face two bad extremes

The first extreme is overclaiming. A metric moved during the pilot, so the product receives full credit. The result enters the sales deck, case study, or investor update without a serious look at alternative explanations.

The second extreme is paralysis. Because perfect attribution is impossible, the team refuses to learn from operational evidence until it can run an academic-quality study.

Both approaches waste the pilot.

The founder’s job is to make a bounded decision under uncertainty. Should we stop, redesign, extend, convert to paid use, or expand? The evidence standard should match that decision.

A small internal pilot may justify the next test without justifying a public market claim. A controlled operational result may justify paid continuation without proving that the effect will generalize across every customer. Evidence can be useful without being universal.

Start with the decision, not the metric

Before asking whether the pilot succeeded, define the decision it was meant to support.

Are we testing whether users can complete the workflow? Whether the output meets a quality threshold? Whether managers will review exceptions? Whether the organization can provide the required data? Whether the buyer will pay to continue after the pilot?

These questions require different evidence.

The pilot brief should define the decision, owner, scope, baseline, success evidence, review cadence, and conversion path before the work begins. That is the core argument of A Pilot Needs A Conversion Contract. The counterfactual adds another requirement: what comparison will help us interpret the observed result?

The baseline is the first counterfactual

The simplest comparison is the workflow before the pilot.

Measure the relevant period using the same definition you will use during the pilot. If the pilot tracks time to follow up, define when the clock starts and stops. If it tracks accepted outputs, define who accepts them and what standard they apply. If it tracks CRM completeness, specify the required fields and the observation window.

A vague baseline creates a convenient story. A defined baseline creates a comparison.

The baseline should also include variation. One unusually bad week can make any intervention look effective. Several weeks, representative cases, or a range of performance provide a more honest reference.

Use a comparable window

The before and after periods should be similar enough to interpret.

Were call volumes comparable? Did the team handle the same kind of accounts? Were the same people involved? Did seasonality, a campaign, a pricing change, or a staffing event alter the work? Was the pilot week unusually quiet or unusually supervised?

The goal is not to eliminate every difference. It is to name the differences that could plausibly explain the outcome.

Sometimes the correct conclusion is: performance improved during the pilot, but the comparison window was favorable, so we need another period before attributing the result.

That is not a weak result. It is a precise one.

Compare parallel workflows when possible

If the organization has similar teams, regions, queues, or workflow types, one can act as a comparison while the other uses the new system.

The groups do not need to be identical, but the differences should be visible. A phone-heavy sales team handling inbound leads should not be compared casually with a strategic account team working long cycles.

Parallel comparison is especially useful when external conditions change for everyone. If both groups face the same seasonal shift but only one shows the operational improvement, the case for product contribution becomes stronger.

Look for the mechanism

A metric change is more credible when the team can explain how the product produced it.

Suppose follow-up completion improves. Did the system identify promised actions from calls? Did it assign an owner? Did it create a draft? Did a manager review an exception queue? Which step changed behavior?

The mechanism creates a chain:

Product action -> workflow behavior -> intermediate signal -> business outcome

If the business outcome moves but the expected workflow behavior does not, the product may not be the cause. If the intermediate signals move in the predicted direction, the attribution becomes more plausible.

This is one reason a pilot should instrument the workflow, not only the final KPI.

Track what changed around the product

Pilots rarely introduce only software. They also introduce attention, training, process clarification, management review, and new definitions of good work.

Those changes are not noise to ignore. They may be part of the actual solution.

If a review cadence is necessary for the product to create value, include it in the operating model. If the benefit disappears without an implementation specialist manually correcting every output, the product may not yet be ready to scale economically.

The founder needs to distinguish product contribution from pilot support, then decide which support belongs in the repeatable offer.

Name alternative explanations

At the review, ask the team to list the strongest plausible reasons for the result besides the product:

  • workload or case mix changed;
  • users were more attentive because they were observed;
  • a manager intervened more frequently;
  • another tool or process changed;
  • the baseline was incomplete;
  • only the easiest cases entered the pilot;
  • problematic outputs were silently excluded;
  • the review standard changed during the test.

Then look for evidence that strengthens or weakens each explanation.

The purpose is not to argue against the product. It is to find the next uncertainty worth testing.

Use contribution language

Founders often feel pressure to translate every result into a clean claim. Better language preserves the evidence level.

Instead of “The product reduced follow-up time by 40 percent,” the supported statement may be: “During a four-week pilot in one team, median follow-up time was 40 percent lower than the preceding comparison period. The pilot also introduced weekly manager review, so the result reflects the combined workflow rather than software alone.”

That sentence is less dramatic and more useful. It tells a buyer or investor what happened, where, compared with what, and under which limitations.

Case studies need evidence levels because discovery evidence, prototype behavior, operational pilot results, and production economics support different decisions.

A Sales Black Box pilot example

For a Sales Black Box pilot, I would not treat a better CRM completion rate as sufficient evidence by itself.

I would want to know the pre-pilot completion rate, the call types included, whether the same fields were required, how often humans corrected suggested updates, and whether managers changed their review behavior. I would also track the mechanism: calls captured, actions identified, drafts accepted, exceptions reviewed, and follow-ups completed.

If follow-up improves while capture and review operate as designed, the case for contribution becomes stronger. If managers manually rescue every action, we have still learned something valuable: the workflow may work, but the product or implementation model needs another iteration before scale.

This is how I want founder judgment to work. Not “Did the graph go up?” but “What do we now believe, how strongly, and what decision does that permit?”

Use a counterfactual ladder

Not every pilot can support the same level of comparison. A practical ladder is:

  1. Documented baseline: compare the workflow before and during the pilot.
  2. Matched time window: control obvious differences in volume, team, and case mix.
  3. Parallel comparison: compare a similar workflow not using the intervention.
  4. Staggered rollout: introduce the system at different times across groups.
  5. Controlled experiment: use randomization or another stronger design where feasible and proportionate.

Move up the ladder when the decision risk justifies the extra effort. Expanding a pilot to one adjacent team may require less certainty than making a major investment or publishing a strong causal claim.

The HM Treasury Magenta Book offers a deeper framework for evaluation and counterfactual reasoning. Founders do not need to reproduce government evaluation practice, but the principle transfers: observed change must be interpreted against a credible account of what would otherwise have happened.

Turn uncertainty into the next decision

At the end of the review, place the result into one of four paths:

  • Strong enough to convert or expand: the outcome, mechanism, and comparison support the next commitment.
  • Promising but bounded: continue in a defined scope to test a remaining attribution question.
  • Workflow works, product needs redesign: the operating model creates value, but automation, usability, or economics are not ready.
  • No meaningful contribution: stop or revisit the underlying problem.

The decision should also update how the result can be described publicly. A promising pilot is not a production case study. An operational result is not repeatable economics. A paid continuation is meaningful commercial evidence, but it still may not prove broad retention.

Review one pilot before making the story cleaner

Take the strongest pilot result currently being discussed and write four lines:

Observed outcome

Comparison or counterfactual

Alternative explanations

Decision this evidence supports

If the second line is empty, do not discard the result. Improve the review design before turning it into a stronger claim.

Proof Engine’s Pilot-to-Revenue Program is built around this transition: define what the pilot must prove, collect evidence that supports a real decision, and connect validated value to a paid path.

As a founder, I do not need every pilot to prove the entire business. I need it to reduce the right uncertainty without lying to me about what caused the result.

FAQ

What is a counterfactual in a pilot review?

An answer to the question of what would probably have happened without the intervention. It does not require a randomized trial. It requires enough comparison discipline to keep a positive result from turning into a claim the evidence cannot support.

What is the simplest counterfactual a founder can use?

The baseline: the same workflow before the pilot, measured with the same definition, including its normal variation rather than one unusually bad week. Pair it with a comparable window, and name differences in volume, team, case mix, or seasonality that could explain the outcome.

What is the counterfactual ladder?

Five levels of comparison: a documented baseline, a matched time window, a parallel comparison with a similar workflow, a staggered rollout, and a controlled experiment. Move up the ladder when the decision risk justifies the extra effort, such as a major investment or a strong public causal claim.

How should a pilot result be described when the evidence is limited?

With contribution language: what happened, where, compared with what, and under which limitations, including support that came with the pilot such as extra manager review. That keeps the evidence level honest and tells a buyer or investor what the result does and does not support.

Sources