How to Design an AI Pilot That Produces an Honest Decision

An AI pilot should produce evidence for a business decision. It is not a longer demonstration and its purpose is not to make deployment inevitable.

A credible pilot compares the proposed workflow with the current process, uses representative work, tests known failure conditions, measures human effort and applies decision rules agreed before the results are known.

The possible outcomes are continue, change and stop.

The difference between an AI demonstration and a pilot

A demonstration answers a narrow capability question. Can a model extract information from this document, draft this response or use this tool under prepared conditions?

That is useful during discovery, but a project decision needs evidence about the operating process.

A pilot asks whether the whole change works with ordinary inputs, real exceptions, actual users, existing systems, required controls and the review effort the organisation can sustain.

The distinction can be expressed in one line:

A demonstration tests an example. A pilot tests a decision.

Treating a polished demonstration as a pilot removes the cases and conditions most likely to change the investment decision.

Start with a falsifiable statement

Write a hypothesis that connects the proposed intervention to the intended business result and names what must not get worse.

For example:

AI assistance will reduce the median time from a qualified enquiry to an approved, customer-ready proposal, without increasing pricing errors, unauthorised commitments, senior review time or customer corrections.

The organisation should replace the general wording with its own baseline, target, scope and risk thresholds.

A strong pilot hypothesis includes:

• A defined unit of work
• The current baseline
• The proposed change
• An observable outcome
• Quality and risk boundaries
• The users and cases included
• A period or volume sufficient to test the expected variation

Avoid choosing a sample size through guesswork. Base it on the frequency of important case types and exceptions, then obtain statistical or evaluation advice where the decision needs it.

Write the decision rules first

Before building or configuring the system, define the evidence for each outcome.

Continue

The new workflow meets the required outcome, quality and control thresholds; human effort and operating cost remain acceptable; and the evidence covers the intended case types.

Change

The test identifies a bounded weakness that could be addressed by narrowing scope, changing the workflow, improving an input, adding a control or testing another intervention.

Stop

The expected value does not appear, review and correction consume the benefit, serious failures cannot be controlled, necessary data cannot be used, or the operating burden is not justified.

Assign the decision to an operational owner who is independent enough to reject a technically impressive result.

Written rules reduce the temptation to move the threshold after seeing disappointing evidence.

Establish a fair baseline

Measure the present process using the same unit and quality boundary that will apply to the pilot.

For a proposal workflow, the baseline may include:

• Time from qualified enquiry to approved proposal
• Active drafting time
• Time spent gathering missing information
• Review and approval time
• Pricing and scope corrections
• Cases returned or rejected
• Customer-requested changes
• Volume by proposal type

The current process does not need to be good or consistent. It does need to be measured honestly.

When evaluating the new workflow, include all the surrounding work: preparing inputs, checking sources, correcting output, escalating exceptions, approving the result and recovering from failure.

Do not compare a model's generation time with a person's end-to-end task time.

Build a representative evaluation set

Use recent work from the population the system is intended to handle. The evaluation set should contain normal cases and the variation most likely to affect the decision:

• Different case and customer types
• Incomplete, poor and conflicting inputs
• Known exception categories
• Previous errors and corrections
• Cases close to policy or approval boundaries
• Work that should be refused or sent to a person

Record why each case is included and the expected result or assessment method.

Some tasks have a known answer that can be checked. Others require qualified judgement. In those cases, define the review criteria, use more than one reviewer where the risk warrants it and retain disagreements as evidence about the task.

Keep a decision set separate from examples used to develop prompts, rules and workflows. A team that repeatedly adjusts the system against the same cases will learn those cases and overestimate performance on new work.

Move from testing to live use in stages

A staged design limits exposure while the evidence develops.

Stage 1: Offline evaluation

Use historical cases without changing live work. Measure output quality, refusal, routing and performance across case types. Identify missing data and systems before users depend on the result.

Stage 2: Challenge testing

Test the conditions most likely to cause harm or reveal a weak control. These may include conflicting sources, missing evidence, misleading text, approval boundaries, prohibited data and unavailable tools.

The aim is to observe failure and confirm whether the system and the surrounding workflow detect it.

Stage 3: Shadow operation

Run the system with current work while the existing process remains authoritative. Compare results without allowing the AI output to make the live decision.

This stage exposes timing, integration, user and input problems that historical testing can miss.

Stage 4: Bounded live use

Limit live use to named users, suitable case types and agreed actions. Provide a manual route, active monitoring, named reviewers and a tested shutdown method.

Expansion is another decision, not the automatic end of the pilot.

The NIST ARIA pilot evaluation report describes three related evaluation levels: model testing, red teaming and field testing. Together they examine technical performance, adversarial or difficult conditions, and interaction with people in realistic settings.

Measure the workflow rather than one model metric

An AI pilot needs a scorecard that reflects the business result and the operating burden.

Report distributions and case groups where possible. Averages can hide long delays, reviewer differences and small groups of high-consequence failures.

Measure reviewers as part of the system. If users repeat the whole task because they do not trust the output, or approve it without checking because it usually looks right, the workflow has not performed as designed.

The NIST AI Risk Management Framework Core recommends qualitative, quantitative or mixed measurement methods, testing before deployment and at regular intervals, and evaluation under conditions similar to the intended deployment. It also recognises the value of independent, domain and user assessment.

Keep an evidence and change record

Record the version of the model, prompt, tools, data sources, rules and workflow used for each evaluation. Log changes with the reason and the effect observed.

Without that record, a final percentage may combine several different systems and say little about the version proposed for live use.

The pilot report should include failed and excluded cases. It should distinguish development examples from the held-back decision set and show performance by case type.

Independent review can help when the delivery team has a strong interest in proceeding. At minimum, the operational owner and qualified users should be able to inspect the evidence, challenge exclusions and confirm that the test reflects their work.

Use a one-page pilot charter

A short charter can keep the pilot tied to its decision. It should contain:

• The hypothesis and business owner
• The workflow and case types in scope
• The baseline and required measures
• The evaluation and held-back case sets
• The stages and exposure limits
• The human roles and review method
• The important failure conditions and controls
• The continue, change and stop rules
• The system and evidence record
• The decision date and decision owner

The charter is not the full test plan. It is the agreement that prevents the test from becoming an open-ended effort to make the technology look ready.

Stopping can be the successful result

A pilot succeeds when it reduces uncertainty enough to support the next decision.

That may mean proceeding with a limited workflow, redesigning the intervention, completing prerequisite work or stopping the project. Discovering that review consumes the expected saving, that important exceptions cannot be controlled or that the data cannot be used is valuable evidence.

The four decisions in this series form a sequence:

1. Define the operational change before selecting technology.
2. Assess the units of work and choose the right intervention for each.
3. Examine value, risk and organisational readiness together.
4. Test the remaining assumptions through a pilot allowed to say no.

AI earns its place by improving the system under evidence the organisation is prepared to accept.