Catalogue › Opportunity Management › pilot-selection

Pilot selection

Chooses which AI opportunities to pilot and designs each pilot so it can prove or disprove its value hypothesis - a representative cohort, a comparison, a fixed duration, success and stop thresholds set in advance, quality checks, risk containment and an exit plan. Use when an opportunity is ready to test with real users, when pilots keep running indefinitely, or when a pilot ended and nobody could tell whether it worked.

Department

Opportunity Management

Lifecycle

Mobilise, Deliver, Measure

Maturity

explorer, adopter, builder, scaler, ai-enterprise

Risk tiers

0, 1, 2, 3, 4

Platforms

platform-neutral

Editions

starter, scale, enterprise

Reviewers

evaluation-quality, responsible-ai

Status

draft

Copilot Cowork

Opportunity Management plugin for Copilot Cowork, or this skill on its own: Download SKILL.md

Licence

CC BY 4.0, by Simon Foote

Purpose

This skill decides which opportunities are worth piloting and designs each pilot as a fair test of its value hypothesis. A good pilot can fail: it has a representative cohort, a comparison, a fixed end date and thresholds agreed before it starts, so the result is a decision rather than an impression. Pilots usually go wrong by being chosen for enthusiasm rather than learning value, by recruiting only volunteers who would succeed with anything, by having no comparison, or by drifting on for months because nobody set a date to decide.

When to use

• A prioritised opportunity is ready to be tested with real users before wider investment.
• Several candidates are competing for limited pilot capacity.
• Previous pilots ran indefinitely or ended without a clear result.
• A sponsor wants to "try" a solution and needs a test that will actually inform a decision.
• A licensed capability is being trialled with a group before broader rollout.

When not to use

• Writing the value hypothesis the pilot will test - use opportunity-management:value-hypothesis.
• Deciding after the pilot whether to scale, iterate, pivot or stop - use opportunity-management:scale-or-stop.
• Testing an agent's quality before any users see it - use m365-agents:agent-testing.
• Planning a broad Copilot launch to a business unit - use copilot-adoption:launch-plan.
• Building a throwaway prototype to explore feasibility - use copilot-code:ai-assisted-prototyping.

Questions to ask first

1. Which hypothesis will the pilot test, and is its disproof condition agreed?
2. Who would the solution serve at scale, and can a pilot cohort represent them?
3. What would we compare against: a similar group without the solution, the same group before, or both?
4. How long must the pilot run for the primary measure to move, and when will we decide?
5. What could go wrong for users or customers during the pilot, and how will it be contained?

Required inputs

• The value hypothesis with baseline, range, leading indicators and disproof condition.
• The risk tier and the reviews the tier requires before users are involved.
• Evidence that the solution has passed pre-pilot quality testing appropriate to its pattern.
• An engaged business owner and the managers of the proposed cohort.

A pilot cannot start without a measured baseline and agreed thresholds. Where they are missing, the opportunity returns to opportunity-management:value-hypothesis. Never set or adjust thresholds after the pilot has begun without recording the change and the reason.

Decision framework

Select pilot candidates with these tests. A candidate should pass all five.

Then design the pilot.

Practitioner approach

Choose pilots for what they will teach, not for how likely they are to look good. Ask, for each candidate, what decision the result will inform and what happens if it fails. A pilot whose failure would change nothing is a demonstration, and demonstrations should be labelled as such rather than consuming pilot capacity.

Confirm the hypothesis is ready. The baseline must be measured, the primary measure collectable during the pilot, and the disproof condition agreed. If not, stop here; running a pilot without these produces opinions, and opinions favour whoever shouts loudest at the review.

Design the cohort with managers. Volunteers are useful for early feedback but are a biased sample; the eventual users include sceptics and less confident colleagues. Aim for a mix by role and confidence, and agree with managers that participants have time to learn and use the solution. Keep the cohort large enough that results can be reported at team level without exposing individuals.

Choose the comparison. The strongest is a similar group doing the same work without the solution over the same period. Where that is impractical, use the cohort's own baseline and watch for seasonal or volume changes that could explain the result. Record known confounders before you start.

Set duration and dates. The pilot must be long enough for people to form habits and for the measure to move, and short enough to keep attention. Put the decision meeting in the diary at the start, with the business owner and sponsor committed to attend.

Write the thresholds into the pilot charter: the result that means scale, the result that means iterate, and the result that means stop. Also set quality thresholds with the evaluation route, and agree that a pilot that meets value thresholds but fails quality thresholds does not count as a success.

Contain risk. Limit reach to the cohort, keep human review on outputs where the risk tier requires it, monitor for harm, and make sure the solution can be switched off quickly. Complete required reviews before the first user is involved.

Plan the exit. Users should know what happens if the pilot stops; data created during the pilot needs a retention decision. A pilot that cannot end cleanly will not be allowed to end. Then run it, collect leading indicators weekly, and hand the evidence to opportunity-management:scale-or-stop on the agreed date.

Outputs

• A pilot selection record: candidates considered, tests passed or failed, and the chosen pilots with reasons.
• A pilot charter for each pilot: hypothesis, cohort, comparison, duration and decision date, success, iterate and stop thresholds, quality checks, support, containment, exit plan and owners.
• A weekly leading-indicator summary at team level during the pilot.
• An evidence pack for the scale-or-stop review, feeding the scale-or-stop-review template.

Microsoft platform considerations

The pattern affects what the pilot can measure and contain. Individual Copilot use is visible only through aggregate reporting; agents built in Copilot Studio can be limited to a pilot audience and expose analytics; Foundry solutions can be instrumented in detail. Licence assignment for pilot cohorts and any trial entitlements are confirmed with the licensing guide or a licensing partner, never assumed. Check capability against references/sources.md.

Adoption considerations

Pilot results depend heavily on enablement. Give the cohort learning, a named contact and Champions support, and record the enablement provided so a scaled rollout can repeat it. Under-supported pilots fail for adoption reasons and are wrongly read as solution failures.

Knowledge considerations

Where the solution grounds on organisational knowledge, confirm the knowledge is current before the pilot and freeze or log changes during it, so that results reflect the solution rather than shifting content.

Security and privacy considerations

Complete the security and privacy reviews required by the risk tier before any user is involved. Tell participants what data the pilot collects and why, collect only what the measures need, and agree retention with the data protection officer.

Responsible AI considerations

Participants should know they are in a pilot and how to raise concerns. Where outputs affect customers or decisions about people, keep human review in place and include fairness or quality checks, not only speed. Pilots of tier 3 and 4 solutions need responsible-ai agreement before start.

Evaluation requirements

Agree quality thresholds and sampling with the evaluation-quality department before start, and run them throughout. Value thresholds and quality thresholds must both be met for a pilot to count as a success.

Value measures

The pilot measures the primary measure from the hypothesis against baseline and comparison, plus leading indicators, reported at team level. At portfolio level, measure the share of pilots that reach a decision on the agreed date. The AI portfolio manager owns these; never report individual participants' usage or performance.

Scale notes

Starter

One or two pilots at a time with a small cohort, the cohort's own baseline as comparison, and a decision meeting with the sponsor at a fixed date.

Scale

A pilot charter for every pilot, a comparison group where practical, and pilot capacity limited so that each pilot is properly supported.

Enterprise

Pilots in one unit are designed to be repeatable in others; the hub keeps a library of charters and results, and requires a comparison group for strategic bets.

Hand-offs

• To opportunity-management:scale-or-stop with the evidence pack on the decision date.
• To m365-agents:agent-release-readiness or the equivalent release route before users are involved.
• To copilot-adoption:launch-plan when a successful pilot is scaled to a wider audience.
• Back to opportunity-management:value-hypothesis when a candidate is not yet testable.

Reviewer gates

From risk tier 2, evaluation-quality agrees the quality thresholds and sampling before the pilot starts. From tier 3, responsible-ai must agree the containment and oversight design. Each finding is blocking or advisory; a blocking finding stops the pilot from starting.

Failure modes

• Only enthusiastic volunteers take part, results look excellent, and the scaled rollout disappoints.
• No comparison is set, so a seasonal drop in workload is credited to the solution.
• Thresholds are set after results arrive, and every pilot is declared a success.
• The pilot has no end date and becomes an unowned live service.
• Quality is never checked, and a pilot that "saves time" is quietly producing wrong answers.

Sources

references/sources.md in this skill lists the source records behind this skill, what each is authoritative for, how volatile it is and what you may do with it. Product availability, licensing, limits and preview status change often, so check the current source before stating them, and cite what you used.

Never

• Start a pilot without a measured baseline, agreed thresholds and a decision date.
• Change thresholds during the pilot without recording the change and the reason.
• Report or compare individual participants' usage or performance.
• Involve users before the reviews required by the risk tier are complete.