Catalogue › Microsoft Foundry › model-evaluation

Model evaluation

Runs the comparative evaluation that decides which candidate model, prompt or configuration performs a specific task best in a Microsoft Foundry solution - task-specific test set, quality and tool-calling measures, latency and cost per task, thresholds set before results, repeated runs and a written comparison. Use when a model shortlist needs testing, when a model version is being replaced, or when two configurations must be compared fairly.

Department

Microsoft Foundry

Lifecycle

Deliver, Assure, Improve

Maturity

builder, scaler, ai-enterprise

Risk tiers

1, 2, 3, 4

Platforms

ai-foundry, azure

Editions

starter, scale, enterprise

Reviewers

evaluation-quality, responsible-ai

Status

draft

Copilot Cowork

Microsoft Foundry plugin for Copilot Cowork, or this skill on its own: Download SKILL.md

Licence

CC BY 4.0, by Simon Foote

Purpose

Choosing between models, versions or configurations is an experiment, and it deserves the discipline of one: a fixed task, a representative test set, measures and thresholds agreed in advance, and enough repeated runs to tell a real difference from noise. This skill runs that experiment for a Foundry solution and turns it into a written comparison the selection and release decisions can rely on. Model evaluation goes wrong when a vendor benchmark or a good demonstration stands in for testing, when candidates are run on different prompts or data, and when thresholds are quietly set after the results show which model the team already preferred.

When to use

• ai-foundry:model-selection has produced a shortlist and needs comparative evidence to choose.
• A deployed model version is being retired or upgraded and the replacement must be shown to be at least as good.
• A team wants to swap to a smaller, cheaper or faster model and needs to know what quality is lost.
• Two prompts, retrieval settings or agent configurations need a fair comparison.
• Different steps in a multi-step solution might use different models, and each step needs evidence.

When not to use

• Shortlisting candidates and recording the final choice - use ai-foundry:model-selection.
• Setting the overall evaluation strategy for the solution and its release - that is the reviewer skill evaluation-quality:ai-evaluation-strategy.
• Building the reference dataset and its governance - that is the reviewer skill evaluation-quality:golden-dataset.
• Automating evaluation runs in the build pipeline - use ai-engineering:evaluation-harness.
• Testing harmful-content and prompt attack behaviour - use ai-foundry:foundry-safety-evaluation.

Questions to ask first

1. What exact task is being compared, what does a correct output look like, and who decides when an output is good enough?
2. What are the candidates, and what is held constant across them (prompt, retrieval, tools, settings)?
3. Which measures matter for this decision, quality, tool-calling accuracy, format adherence, latency or cost per task, and what threshold must each meet?
4. Is there a representative test set drawn from real work, and does it include hard, edge and out-of-scope cases?
5. What difference between candidates would actually change the decision?

Required inputs

• The task definition and shortlist from the model-selection-record template.
• A test set of real or realistic inputs with expected outputs or reference answers, confirmed by a business or knowledge owner.
• A scoring rubric for judgement-based measures, ideally from evaluation-quality:ai-quality-rubric.
• Non-production deployments of each candidate, configured as they would run in production.

Where the test set or thresholds are missing, mark them [To confirm: ...] and do not run the comparison. Results without a pre-agreed threshold invite the team to rationalise its favourite.

Decision framework

Choose the cheapest and fastest candidate that passes every threshold. If none passes, the answer is to change the prompt, retrieval or task design, or to stop, not to lower the threshold.

Practitioner approach

Fix the task and the thresholds first and write them down with a date. The business owner agrees what good enough means, in measures they understand, before any candidate is run. Agree too what size of difference would matter; a small gain in quality may not justify a large rise in cost or latency.

Build or borrow a test set that looks like real work. Use genuine examples where privacy allows, or realistic synthetic ones checked by people who know the domain, and include hard cases, edge cases, ambiguous requests and requests the solution should decline. A few dozen well-chosen cases beat hundreds of easy ones. Hold back a portion the team does not tune against, so the final comparison is not flattered by over-fitting.

Hold everything else constant. Each candidate runs with the same prompt structure, retrieval results, tools and settings, adjusted only where a model's documented conventions require it, and those adjustments are recorded. If a prompt has to be re-tuned for a candidate, give every candidate comparable tuning effort, or the comparison measures effort rather than models.

Run each candidate several times. Model outputs vary between runs, so a single run can mislead. Report the spread, not only the average, and treat overlapping results as a tie. Measure latency on deployments configured like production, and calculate cost per completed task rather than per call, because retries and longer outputs change the picture.

Use model graders carefully. An AI-assisted grader speeds up scoring of open answers, but it has its own biases and may favour outputs that resemble its own style. Calibrate it against a sample scored by people before relying on it, and keep people scoring the cases that matter most.

Read the failures, not just the scores. Group wrong outputs by cause, such as missing context, wrong tool, format breakage or over-refusal, because the cause often points to retrieval or prompt fixes that would change the ranking.

Write the comparison into the evidence section of the model-selection-record template: task, test set version, candidates and settings, thresholds and their date, results with spread, failure analysis, recommendation and what the evaluation could not show. Keep the set and scripts so the comparison can be repeated when a model changes.

Outputs

• A comparative evaluation report in the evidence section of the model-selection-record template, with dated thresholds, results and spread per candidate, failure analysis and recommendation.
• A versioned, task-specific test set with a held-back portion, stored with restricted access.
• Evaluation configuration and scripts handed to ai-engineering:evaluation-harness for repeat runs.
• Inputs to the evaluation-plan template for the solution's release evidence.

Microsoft platform considerations

Microsoft Foundry provides evaluation tooling, built-in quality and safety evaluators and ways to compare runs, and the evaluators, supported models and preview status change often. Check the current source records before relying on a built-in evaluator, and calibrate it against human judgement for the task. Third-party models in the catalogue may differ in deployment type, throughput and pricing model, so measure latency and cost on the deployment type that would be used in production rather than comparing list figures.

Adoption considerations

Not applicable to the evaluation itself, because users experience the chosen model rather than the comparison. Where real user examples are used in the test set, tell the business owner how they were chosen and protected.

Knowledge considerations

Where the task relies on retrieved knowledge, freeze the retrieval results or index version used in the comparison so knowledge changes do not masquerade as model differences, and record the version.

Security and privacy considerations

Test sets built from real work can contain personal or confidential data. Minimise, mask or replace it, restrict access to the set and the outputs, and run evaluations only on deployments approved for that data. Data processing questions for third-party models go to the DPO.

Responsible AI considerations

Include cases that represent the full range of users and inputs, such as different languages, writing styles and accessibility needs, and report quality by subgroup where outputs affect people, coordinating with responsible-ai:fairness-review.

Evaluation requirements

This skill is evaluation. Its own quality rests on thresholds dated before results, a held-back set, repeated runs, calibrated graders and a recorded test set version, so the comparison can be reproduced and audited.

Value measures

Track cost and latency savings from evidence-based model choices, and how often post-release quality matches the evaluation, owned by the solution owner. A gap between evaluated and live quality signals an unrepresentative test set.

Scale notes

Starter

Compare two or three candidates on a small, well-chosen set with human scoring and three runs each. Record the result on the model selection record and keep the set for the next model change.

Scale

Maintain task-type test sets and calibrated graders that solutions reuse, and run comparisons through a shared harness so results are comparable across teams.

Enterprise

Run a standing evaluation service that re-tests widely used models when new versions appear, publishes comparisons to solution owners and feeds the approved model list.

Hand-offs

• To ai-foundry:model-selection with the recommendation and evidence.
• To ai-engineering:evaluation-harness to automate repeat runs.
• To ai-foundry:foundry-prompt-engineering or ai-foundry:retrieval-design when failure analysis points there.
• To evaluation-quality:ai-release-evidence as part of the release evidence pack.

Reviewer gates

From risk tier 2, evaluation-quality checks that thresholds predate results, the test set is representative and the comparison is reproducible, and responsible-ai checks subgroup results where outputs affect people. A threshold changed after results is a blocking finding.

Failure modes

• A vendor benchmark replaces testing, and the chosen model underperforms on the organisation's own documents.
• Candidates are run with different prompts or retrieval, and the comparison measures setup rather than models.
• A single run is treated as decisive, and the model chosen on a lucky result performs worse in production.
• An uncalibrated model grader favours verbose answers, and the evaluation rewards the wrong behaviour.

Sources

references/sources.md in this skill lists the source records behind this skill, what each is authoritative for, how volatile it is and what you may do with it. Product availability, licensing, limits and preview status change often, so check the current source before stating them, and cite what you used.

Never

• Set or change a threshold after seeing the results.
• Select a model on vendor benchmarks or a demonstration alone.
• Compare candidates under different prompts, data or settings without recording and justifying the difference.
• Use real personal data in a test set without minimisation, access control and an approved deployment.