Catalogue › Copilot Code › application-validation

Application validation

Validates an application built with Copilot Code against its solution brief, tracing every acceptance criterion to reviewed tests, negative and edge cases, non-functional checks and business owner acceptance, and records a pass, conditional pass or fail. Use when an AI-built application is feature-complete, when a business owner must accept a build, or when generated tests look thorough but nobody has checked them against requirements.

Department

Copilot Code

Lifecycle

Assure, Launch

Maturity

builder, scaler, ai-enterprise

Risk tiers

1, 2, 3, 4

Platforms

copilot-code, github, azure

Editions

starter, scale, enterprise

Reviewers

evaluation-quality

Status

draft

Copilot Cowork

Copilot Code plugin for Copilot Cowork, or this skill on its own: Download SKILL.md

Licence

CC BY 4.0, by Simon Foote

Purpose

When a coding agent writes both the code and the tests, a green test run proves only that the code does what the tests say, and the tests may say the wrong thing. This skill validates an AI-built application against what was actually required: it traces every acceptance criterion in the solution brief to independent evidence, adds negative, edge and non-functional checks, and closes with acceptance by the business owner using real scenarios. It goes wrong when validation is a demonstration of the happy path, or when the generated test suite is accepted as proof without anyone reading it.

When to use

• An AI-built application is feature-complete and needs acceptance before deployment.
• The business owner must formally accept a build.
• The generated test suite looks comprehensive but has never been checked against the brief.
• A significant change has been made to a live AI-built application.

When not to use

• Reviewing individual code changes as they are made - use copilot-code:generated-code-review.
• Security review of the generated code - use copilot-code:generated-code-security-review.
• Checking operational readiness to deploy - use copilot-code:code-deployment-readiness.
• Testing the AI features inside a pro-code AI solution (model quality, groundedness) - use ai-engineering:ai-solution-testing.
• Validating a Copilot Cowork workflow - use copilot-cowork:workflow-validation.

Questions to ask first

1. Does every acceptance criterion in the brief have at least one test or check, and has someone other than the agent confirmed the test asserts the right thing?
2. Which negative and edge cases matter most: wrong user, missing data, bad input, lost connection, duplicate submission?
3. Which non-functional requirements apply (accessibility, performance, availability, retention, logging), and how will each be shown?
4. Who will carry out user acceptance, with which real scenarios, and what threshold will the business owner accept?
5. What has changed since the brief, and was each change agreed?

Required inputs

• The agreed solution brief with acceptance criteria and non-functional requirements.
• The work items and their acceptance tests from copilot-code:requirements-to-build.
• A test environment with synthetic or approved test data, isolated from production.
• A business owner and a few representative users available for acceptance.

If acceptance criteria are missing or were changed without agreement, return to the brief before validating. Never validate with real personal data outside an approved environment.

Decision framework

Build a traceability matrix and judge each criterion.

Practitioner approach

Start with the brief, not the test suite. List every acceptance criterion and non-functional requirement, then find the evidence for each. This order matters: starting from the tests shows what was built, starting from the brief shows what was required, and the gaps between them are the point of validation.

Read the tests. For each criterion, open the test that claims to cover it and check that the assertion actually expresses the requirement. AI-generated tests commonly assert what the code does, test only the happy path, mock away the part that matters, or assert something trivially true. A reviewer who reads assertions will find more problems in an hour than a coverage report will in a week.

Add the negative and edge cases the agent did not think of. Access rules need negative tests from the point of view of each role. Data entry needs invalid, missing and boundary inputs. Integrations need failure cases: the other system is slow, returns an error or returns something unexpected. Offline or retry behaviour needs duplicate and partial-submission cases. These are where AI-built software most often breaks.

Check non-functional requirements explicitly. Run an accessibility check, a basic performance test at expected load, and inspect logs for personal data and useful diagnostics. Confirm data retention behaves as specified. These are rarely covered by generated tests unless the work items demanded them.

Run user acceptance with real scenarios. Ask two to five representative users to complete the journeys in the brief with test data, without coaching. Watch where they hesitate, where the software surprises them, and whether the outcome in the brief is achieved. The business owner then decides whether to accept, using the evidence rather than a demonstration.

Record the matrix, the verdict and any conditions. Keep the reviewed tests as a regression suite so the same evidence can be regenerated after changes. When the agent makes later changes, re-run the suite and re-check the assertions of any tests it modified, because a changed test can quietly redefine "correct".

Outputs

• A traceability matrix from each acceptance criterion and non-functional requirement to reviewed evidence.
• A validation verdict (pass, conditional pass, fail) with conditions, owners and dates.
• A user acceptance record signed by the business owner.
• A reviewed regression suite for future changes.

Microsoft platform considerations

Test environments, pipelines and test tooling sit on the organisation's development and hosting platforms (for example Azure and the repository's pipeline service). Confirm current platform capabilities and limits from the relevant source records. Validation environments must be isolated from production and use synthetic or approved data.

Adoption considerations

User acceptance is the first adoption moment. Users who test the software and see their feedback acted on become its advocates; record their feedback and tell them what changed.

Knowledge considerations

Not applicable as a primary concern, because validation tests software behaviour; where the application surfaces reference content, confirm the content source named in the brief is the one used.

Security and privacy considerations

Negative access tests are part of validation, but they do not replace copilot-code:generated-code-security-review. Check logs and error messages for leaked personal data or internal details. Use only synthetic or approved test data.

Responsible AI considerations

Where the application supports decisions about people, validate that the human decision step exists, is clear and cannot be bypassed, and that users can tell what is automated. Route material concerns to the responsible-ai department.

Evaluation requirements

This skill is the evaluation for AI-built applications. From risk tier 2, align the evidence standard with evaluation-quality:ai-release-evidence so validation results form part of the release evidence pack.

Value measures

Track at team level defects found in validation versus after release, and the share of criteria with reviewed evidence. Use the brief's outcome measure as the baseline for post-release value. Never use validation findings to judge individual engineers.

Scale notes

Starter

A simple matrix in a spreadsheet, reading the key tests, a handful of negative cases, and acceptance by the business owner with two or three users.

Scale

Use a standard traceability template, require reviewed negative tests for every access rule, and keep regression suites in the pipeline.

Enterprise

Integrate validation evidence into the release evidence pack, with independent validation for builds at risk tier 3 and above.

Hand-offs

• To copilot-code:code-deployment-readiness with the verdict and regression suite.
• Back to copilot-code:requirements-to-build with gaps needing new work items.
• To evaluation-quality:ai-release-evidence from risk tier 2.

Reviewer gates

From risk tier 2, evaluation-quality reviews the traceability matrix and evidence quality before release. Findings are blocking or advisory; an unproven access, data integrity or safety criterion is blocking.

Failure modes

• The generated test suite is accepted as proof without anyone reading the assertions.
• Validation demonstrates the happy path, and the first missing-data case fails in production.
• Non-functional requirements are assumed rather than checked, so logs fill with personal data.
• A later change by the agent modifies a test, quietly redefining what counts as correct.

Sources

references/sources.md in this skill lists the source records behind this skill, what each is authoritative for, how volatile it is and what you may do with it. Product availability, licensing, limits and preview status change often, so check the current source before stating them, and cite what you used.

Never

• Accept generated tests as evidence without a person checking them against the requirement.
• Pass an application with unproven access or data integrity criteria.
• Validate with real personal data outside an approved environment.
• Replace business owner acceptance with a demonstration.