Catalogue › Evaluation and Quality › ai-quality-rubric

AI quality rubric

Writes and reviews the quality rubric used to score AI outputs - dimensions with plain definitions, anchored scoring levels, zero-tolerance failures, pass thresholds agreed with the business owner before results, and calibration of human and AI-assisted graders - with findings classed blocking or advisory. Use when evaluators disagree on scores, when thresholds are being set, or when a rubric is submitted to the gate.

Department

Evaluation and Quality

Lifecycle

Deliver, Assure, Improve

Maturity

adopter, builder, scaler, ai-enterprise

Risk tiers

1, 2, 3, 4

Platforms

m365-agents, copilot-cowork, copilot-studio, ai-foundry, copilot-code

Editions

starter, scale, enterprise

Reviewers

evaluation-quality

Status

draft

Copilot Cowork

Evaluation and Quality plugin for Copilot Cowork, or this skill on its own: Download SKILL.md

Licence

CC BY 4.0, by Simon Foote

Purpose

Two reviewers looking at the same AI answer will often disagree on whether it is good, and an AI-assisted grader will disagree with both in ways nobody has measured. A rubric turns "good enough" into dimensions with plain definitions, anchored scoring levels, a short list of failures that are never acceptable and thresholds the business owner agrees before seeing results. This reviewer skill writes rubrics and reviews those submitted by teams, classifying each finding as blocking or advisory. Rubrics go wrong when the levels are vague ("mostly accurate"), when there are so many dimensions that scoring becomes guesswork, or when the threshold is chosen after the results are in.

When to use

• An evaluation plan names quality dimensions and they need scoring scales and thresholds.
• Human evaluators disagree on scores for the same outputs.
• A team proposes AI-assisted grading and needs a rubric the grader and humans can both apply.
• A rubric and threshold are submitted to the evaluation-quality gate.
• A business owner asks what pass rate is acceptable and wants help deciding.

When not to use

• Choosing which dimensions and methods the solution needs at all - use evaluation-quality:ai-evaluation-strategy.
• Building the items to be scored - use evaluation-quality:golden-dataset.
• Detailed checks of whether claims are supported by sources and citations - use evaluation-quality:groundedness-testing.
• Measuring the business effect of AI on quality of work, rework and error rates - use value-realisation:quality-measurement.
• Assessing people's skills or evidence for learning - use learning-and-skills:learning-evidence-rubric.

Questions to ask first

1. For each dimension, what does an unacceptable, acceptable and excellent output look like, in an example the business owner recognises?
2. Which failures must count as an automatic fail regardless of the overall score?
3. Who will score - subject experts, users, an AI-assisted grader or a mix - and how will their agreement be checked?
4. What pass rate per dimension would the business owner accept for release, and why that figure?
5. Does any dimension need to be scored separately for different user groups or content types?

Required inputs

• The evaluation plan with dimensions, risk tier and consequence statement.
• A sample of real outputs from the solution, including some poor ones, to anchor the levels.
• The business owner, available to agree thresholds before results are seen.
• Behaviour rules from the brief, including refusal and tone requirements.

Where no real outputs exist yet, draft anchors from the brief and mark them provisional; recalibrate after the first run before any threshold decision.

Decision framework

Use few dimensions with three or four anchored levels. More levels add disagreement, not precision.

Add dimensions only when the evaluation plan requires them, such as action correctness or consistency across runs.

Thresholds and zero-tolerance rules:

When reviewing a submitted rubric, classify each finding.

Practitioner approach

Write the rubric with real outputs on the table. Take a handful of answers from the solution, including clearly poor ones, and ask the business owner and a subject expert to sort them into fail, partial and pass for each dimension. Turn their reasons into the anchor wording. Anchors written in the abstract drift; anchors written from examples hold.

Keep the scoring unit clear. Decide whether a multi-part answer is scored as a whole or per part, whether repeated runs of the same item are scored separately, and how a correct answer with a wrong citation is treated. Write each rule into the rubric so two graders reach the same result.

Calibrate the graders before they score alone. Give every human grader the same small sample, compare scores, discuss the disagreements and tighten the anchors where they caused confusion. Repeat until agreement is good enough for the risk tier, and record the result. If an AI-assisted grader will be used, give it the same rubric and compare its scores with the calibrated human scores. Record where it is lenient or harsh. Use it to scale review only where its disagreement is understood, and keep a human-scored sample in every run.

Set thresholds with the business owner, per dimension, before results. Anchor the conversation in the consequence statement: if wrong answers are always checked by an expert, a lower correctness threshold may be acceptable; if users act directly on answers, it must be higher. Do not import thresholds from another solution without that conversation. Agree the zero-tolerance list in the same meeting and record the date.

Present results per dimension, with the zero-tolerance breaches listed separately and the grader agreement shown alongside. A pass rate without the agreement figure invites false confidence.

Revisit the rubric when the solution's purpose or audience changes, when graders start disagreeing again, or when live feedback shows the rubric is passing answers users find unhelpful. Version it alongside the golden dataset so results remain comparable.

Outputs

• A quality rubric: dimensions, plain definitions, anchored levels with examples, scoring rules and zero-tolerance list, recorded in the evaluation-plan.
• Thresholds per dimension, signed and dated by the business owner before the first scored run.
• A calibration record for human graders and any AI-assisted grader.
• A rubric review record with findings classified blocking or advisory, the fix and an owner.

Microsoft platform considerations

Copilot Studio and Microsoft Foundry offer evaluation features that can apply criteria or AI-assisted graders to outputs; what they measure and how they are configured changes over time. Map platform metrics to the rubric's dimensions explicitly, confirm current behaviour in ms-copilot-studio-guidance and ms-foundry-models, and do not assume a platform metric means the same as a rubric dimension of the same name.

Adoption considerations

Share the rubric's plain definitions with users at launch. When people know what "good" means for the solution, their feedback becomes far more useful.

Knowledge considerations

Groundedness and correctness anchors depend on which sources are authoritative. Name them in the rubric so graders do not reward answers drawn from superseded or unofficial content.

Security and privacy considerations

The safety dimension must include exposure of restricted or personal content as a zero-tolerance failure from tier 2, aligned with security-privacy:data-exposure-review. Graders see real outputs, so limit grading to people entitled to the underlying content.

Responsible AI considerations

Where outputs affect people, add a dimension or zero-tolerance rule agreed with responsible-ai:fairness-review, and score samples across the groups affected. Rubrics measure outputs, not individuals; never adapt one to score staff.

Evaluation requirements

This skill defines how evaluation is scored. Minimum from tier 2: anchored levels per dimension, per-dimension thresholds dated before results, a zero-tolerance list, and a calibration record for every grader whose scores count as evidence.

Value measures

Track grader agreement over time, the share of releases where thresholds were agreed before results, and the rate of live complaints about answers the rubric passed. The AI Evaluation Lead owns these measures at solution level; never report individual graders' accuracy.

Scale notes

Starter

Three dimensions with three levels, one example per level, and thresholds agreed by email with the business owner before testing. Two people calibrate on ten outputs.

Scale

A shared rubric library by solution type with standard anchors, local examples and a calibration step at the start of each evaluation cycle. AI-assisted grading introduced only with a recorded comparison.

Enterprise

Central rubric standards owned by the evaluation-quality department, federated graders trained to them, periodic cross-unit calibration and a recorded comparison for every AI-assisted grader in use.

Hand-offs

• To m365-agents:agent-testing, copilot-studio:studio-testing-and-evaluation or ai-engineering:evaluation-harness to apply the rubric.
• To evaluation-quality:groundedness-testing and evaluation-quality:ai-safety-testing for specialist dimensions.
• To evaluation-quality:ai-release-evidence with the thresholds and calibration record.

Reviewer gates

This skill operates within the evaluation-quality gate. From risk tier 2 a rubric with blocking findings cannot be used to support release; below tier 2 findings are advisory. Disagreement over thresholds escalates to the accountable executive for written risk acceptance; the maker cannot lower a threshold.

Failure modes

• Levels described by adjectives only, so graders score by mood and results cannot be repeated.
• A blended score lets an unsafe or ungrounded answer pass because the others were good.
• An AI-assisted grader is lenient on fluent but wrong answers, and nobody checked.
• The threshold is lowered quietly after the first run to match the result.

Sources

references/sources.md in this skill lists the source records behind this skill, what each is authoritative for, how volatile it is and what you may do with it. Product availability, licensing, limits and preview status change often, so check the current source before stating them, and cite what you used.

Never

• Accept a threshold set or changed after results were seen.
• Combine dimensions into a single score that lets a safety or grounding failure pass.
• Use an AI-assisted grader as evidence without a recorded comparison against human scores.
• Use a solution rubric to score or rank the people who built or tested it.