Catalogue › AI Engineering › evaluation-harness

Evaluation harness

Builds the automated evaluation harness for a custom AI solution - dataset loading, repeatable runs against pinned models and prompts, code-based and model-graded evaluators, scoring against agreed thresholds, results history and reports - so quality is checked on every change. Use when an AI solution needs repeatable evaluation in its pipeline, when manual spot checks no longer scale, or when evaluation results cannot be reproduced.

Department

AI Engineering

Lifecycle

Deliver, Assure, Improve

Maturity

builder, scaler, ai-enterprise

Risk tiers

1, 2, 3, 4

Platforms

ai-foundry, azure, github, copilot-code

Editions

starter, scale, enterprise

Reviewers

evaluation-quality

Status

draft

Copilot Cowork

AI Engineering plugin for Copilot Cowork, or this skill on its own: Download SKILL.md

Licence

CC BY 4.0, by Simon Foote

Purpose

Evaluation strategy says what quality means and how much is enough; the harness is the code that measures it, the same way, every time something changes. Without one, teams test by trying a few questions by hand before each release, and regressions reach users. This skill builds the harness: loading the dataset, running the solution under pinned conditions, applying evaluators, scoring against thresholds, storing results and reporting them where decisions are made. It goes wrong when the harness tests a different configuration from production, when model-graded scores are trusted without calibration, and when thresholds are adjusted after seeing the results.

When to use

• A custom AI solution needs automated, repeatable evaluation in its pipeline.
• Manual spot checks before release no longer cover the range of inputs or changes.
• Evaluation results differ between runs and nobody can tell whether quality changed.
• The team is changing a model, prompt, tool or retrieval setting and needs evidence that nothing regressed.
• The evaluation strategy and dataset exist, and someone now has to make them run.

When not to use

• Deciding what to evaluate, which dimensions matter and how evaluation is governed - use evaluation-quality:ai-evaluation-strategy.
• Building and curating the golden dataset itself - use evaluation-quality:golden-dataset.
• Comparing candidate models for a task - use ai-foundry:model-evaluation.
• Setting quality rubrics and acceptance thresholds - use evaluation-quality:ai-quality-rubric.
• Testing a Copilot Studio agent - use copilot-studio:studio-testing-and-evaluation.

Questions to ask first

1. What evaluation strategy, rubric and thresholds have been agreed, and by whom?
2. Where is the evaluation dataset, how is it versioned and does it contain sensitive data?
3. Which dimensions can be checked by code, and which need a model-graded or human judgement?
4. Which changes must trigger a run (prompt, model, tool, retrieval, code), and how long can a run take in the pipeline?
5. Who reads the results and makes the release decision?

Required inputs

• The evaluation strategy and rubric, with thresholds agreed before results are seen.
• A versioned evaluation dataset, or a reference to an immutable version in a governed store.
• A way to run the solution non-interactively against an input, with the same configuration as the target environment.
• Budget and quota for model calls made during evaluation.

If thresholds have not been agreed, build the harness to report scores but do not let it pass or fail a release until the business owner agrees thresholds. Never set thresholds by looking at the first results.

Decision framework

Choose the evaluator for each quality dimension by the cheapest reliable method.

Set run tiers. A fast subset runs on every pull request; the full set runs before promotion and on a schedule.

Practitioner approach

Start from the agreed strategy and rubric. The harness implements decisions; it does not make them. List each quality dimension, its threshold, any zero-tolerance failures and the evaluator chosen for it. Confirm the business owner agreed the thresholds before any result was seen.

Make runs reproducible. Pin the model deployment and version, the prompt version, tool versions and retrieval configuration, and record them with every run. Set generation parameters to the values used in production, not idealised ones. Where outputs vary between runs, run each case more than once and report the spread, so a release decision is not made on noise.

Test the system the user gets. Call the solution through the same orchestration, tools and retrieval it uses in production, against a dedicated evaluation environment with representative data and permissions. Evaluating the model alone, with a hand-built prompt, measures something users never see.

Prefer code evaluators. Where a correct answer is known, check it with code: field values, numbers, dates, the tool called and its arguments, the presence of a refusal. Code evaluators are cheap, fast and stable. Use model-graded evaluators for judgements code cannot make, with a fixed rubric, a pinned grading model and examples. Calibrate them: have people label a sample and report agreement. A grader that disagrees with people is not evidence.

Keep zero-tolerance cases separate. Exposure of restricted data, unsafe content or an unconfirmed write fails the run regardless of average scores. Report them individually.

Store results as history, not just a pass or fail. Keep per-case results, scores, configuration and cost for each run, so trends and the effect of each change are visible. Produce a short report for the pull request and a fuller one for release, showing scores against thresholds, failures by category and changes since the last run.

Wire it into the pipeline with ai-engineering:ai-ci-cd: a fast subset on every change to behaviour-changing assets, the full set before promotion and on a schedule to catch provider-side drift. Control cost by sizing the subset and caching unchanged results where valid. When a failure is found in production, add the case to the dataset through the golden dataset process so it is tested from then on.

Outputs

• An evaluation harness in the solution repository: dataset loader, runner, evaluators, scoring, thresholds configuration, results store and report generator.
• Run tiers and triggers (pull request subset, full pre-release run, scheduled run).
• Calibration results for each model-graded evaluator.
• A results history with configuration recorded per run, feeding evaluation-quality:ai-release-evidence.

Microsoft platform considerations

Microsoft Foundry provides evaluation capabilities, built-in evaluators including quality and safety evaluators, and SDKs that can run in pipelines; the available evaluators, their behaviour and their costs change. Check current capabilities in references/sources.md, pin evaluator versions where possible, and treat built-in evaluators like any model-graded evaluator: calibrate before relying on them.

Adoption considerations

Engineers run the harness when it is fast and its output is useful in the pull request. Keep the subset quick, the report short and the failing cases easy to inspect. Business owners trust results they helped define; show them the report format early.

Knowledge considerations

Where the solution grounds on knowledge, the harness depends on that knowledge being stable in the evaluation environment. Snapshot or version the evaluation corpus, and re-baseline when the knowledge owner makes major changes.

Security and privacy considerations

Evaluation datasets and results can contain personal or confidential data. Hold them in governed stores with access limited to the team, avoid printing sensitive content in pipeline logs, and run evaluation in an environment with the same data protections as production.

Responsible AI considerations

Include cases that reflect the full range of users and inputs, including languages, accessibility needs and groups the solution could treat unfairly, and report results by segment where the strategy calls for it. Safety evaluation results go to the responsible-ai and evaluation-quality reviewers.

Evaluation requirements

The harness itself is tested: evaluators have unit tests on known good and bad outputs, model-graded evaluators have calibration evidence, and a deliberately degraded configuration must fail the run.

Value measures

Track regressions caught before release, regressions found in production, run time and evaluation cost per run. The AI engineering lead owns these; evaluation quality reviewers use them to judge the harness's coverage.

Scale notes

Starter

Begin with a small dataset, code evaluators for the dimensions that allow it, one calibrated model-graded evaluator, and a run before each release. Store results in the repository or a simple store.

Scale

Run subsets on every pull request and full sets before promotion, share evaluators across solutions as a versioned package, and keep results history in a common store.

Enterprise

Provide a shared evaluation platform with standard evaluators, calibration records, scheduled drift runs across solutions and results feeding release evidence and portfolio reporting.

Hand-offs

• To ai-engineering:ai-ci-cd to run the harness as a pipeline gate.
• To evaluation-quality:ai-release-evidence with results for release decisions.
• To evaluation-quality:regression-evaluation for regression policy after changes.
• To evaluation-quality:golden-dataset with production failures to add as cases.

Reviewer gates

From risk tier 2, evaluation-quality reviews the harness: that it tests the production configuration, that evaluators are calibrated and that thresholds were agreed in advance. Thresholds changed after results were seen are a blocking finding.

Failure modes

• The harness evaluates the model with a hand-written prompt, not the deployed solution, and passes a broken release.
• A model-graded evaluator is never calibrated and scores fluent but wrong answers highly.
• Thresholds are lowered after a disappointing run, and the release goes ahead on a moved goalpost.
• Variance between runs is ignored, and a release is blocked or passed on noise.

Sources

references/sources.md in this skill lists the source records behind this skill, what each is authoritative for, how volatile it is and what you may do with it. Product availability, licensing, limits and preview status change often, so check the current source before stating them, and cite what you used.

Never

• Set or change a threshold after seeing the results it will judge.
• Rely on a model-graded evaluator that has not been calibrated against human judgement.
• Evaluate a configuration that differs from the one being released.
• Let average scores hide a zero-tolerance failure.