Catalogue › Evaluation and Quality

CopilotForge - Evaluation and Quality

Use it in Copilot Cowork: download the Evaluation and Quality plugin for Copilot Cowork and upload it under Customize > Plugins > Upload plugin. How to install.

Reviewer-class department that defines evaluation sets, acceptance criteria, failure analysis and release evidence for probabilistic systems.

Layer: assurance (reviewer-class: may block)
Lead role: AI Evaluation Lead
Plugin name: copilotforge-evaluation-quality

Skill Status Lifecycle Description

ai-evaluation-strategy draft Envision, Deliver, Assure, Improve Sets and reviews the evaluation strategy for a probabilistic AI solution - which quality dimensions matter, which methods and datasets measure them, thresholds agreed before results and when evaluation repeats - classifying gaps as blocking or advisory. Use when a tier 2 or higher agent, workflow or Foundry solution enters design, or when nobody can say how the team will know it is good enough.

ai-quality-rubric draft Deliver, Assure, Improve Writes and reviews the quality rubric used to score AI outputs - dimensions with plain definitions, anchored scoring levels, zero-tolerance failures, pass thresholds agreed with the business owner before results, and calibration of human and AI-assisted graders - with findings classed blocking or advisory. Use when evaluators disagree on scores, when thresholds are being set, or when a rubric is submitted to the gate.

ai-release-evidence draft Assure, Launch Assembles and reviews the evaluation evidence pack an AI solution needs at its release gate - plan, dataset and rubric versions, thresholds dated before results, results per dimension, groundedness, safety and red-team outcomes, known limits and monitoring baseline - and gives the evaluation-quality verdict with findings classed blocking or advisory. Use when a tier 2 or higher solution or material change seeks release.

ai-safety-testing draft Deliver, Assure, Improve Tests and reviews whether an AI solution avoids harmful outputs for its audience - hateful, violent, sexual or self-harm content, dangerous or regulated advice, demeaning statements about people, and poor handling of users in distress - checking platform content controls as configured, classing findings blocking or advisory. Use when a solution reaches the public or a broad workforce, or after harmful output is reported.

golden-dataset draft Deliver, Assure, Improve Builds, maintains and reviews the golden dataset an AI solution is judged against - real questions or tasks from users, expected answers and sources confirmed by knowledge owners, edge, refusal and adversarial cases, versioning and safe handling of test data - with findings classed blocking or advisory. Use when evaluation needs a trusted reference set, when expected answers have drifted, or when a test set looks too friendly.

groundedness-testing draft Deliver, Assure, Improve Tests and reviews whether an AI solution's answers are supported by the permitted sources it cites - claim-level support, citation accuracy, behaviour when knowledge is silent, stale or contradictory, and blending with model or web knowledge - classing findings blocking or advisory. Use when a knowledge-grounded agent or retrieval solution approaches release, after knowledge changes, or when users report invented or wrongly cited answers.

red-team-plan draft Assure, Launch, Improve Plans and reviews a red-team exercise for an AI solution - objectives, scope, harm and misuse scenarios, a team with varied perspectives, rules of engagement, safe test environments, evidence recording, tester welfare and follow-through to fixes and re-tests - classing plan gaps and findings blocking or advisory. Use when a tier 3 or 4 solution nears release, when it gains new actions or audiences, or after a serious incident.

regression-evaluation draft Deliver, Assure, Improve Decides and reviews what must be re-evaluated after a change to a live AI solution - instructions, knowledge, actions, model or platform updates - selecting the regression subset, comparing with the last baseline while allowing for normal output variation, and recommending release, hold or rollback, with findings classed blocking or advisory. Use when a change is ready to deploy or Microsoft announces a model or platform change.

Skill	Status	Lifecycle	Description
`ai-evaluation-strategy`	draft	Envision, Deliver, Assure, Improve	Sets and reviews the evaluation strategy for a probabilistic AI solution - which quality dimensions matter, which methods and datasets measure them, thresholds agreed before results and when evaluation repeats - classifying gaps as blocking or advisory. Use when a tier 2 or higher agent, workflow or Foundry solution enters design, or when nobody can say how the team will know it is good enough.
`ai-quality-rubric`	draft	Deliver, Assure, Improve	Writes and reviews the quality rubric used to score AI outputs - dimensions with plain definitions, anchored scoring levels, zero-tolerance failures, pass thresholds agreed with the business owner before results, and calibration of human and AI-assisted graders - with findings classed blocking or advisory. Use when evaluators disagree on scores, when thresholds are being set, or when a rubric is submitted to the gate.
`ai-release-evidence`	draft	Assure, Launch	Assembles and reviews the evaluation evidence pack an AI solution needs at its release gate - plan, dataset and rubric versions, thresholds dated before results, results per dimension, groundedness, safety and red-team outcomes, known limits and monitoring baseline - and gives the evaluation-quality verdict with findings classed blocking or advisory. Use when a tier 2 or higher solution or material change seeks release.
`ai-safety-testing`	draft	Deliver, Assure, Improve	Tests and reviews whether an AI solution avoids harmful outputs for its audience - hateful, violent, sexual or self-harm content, dangerous or regulated advice, demeaning statements about people, and poor handling of users in distress - checking platform content controls as configured, classing findings blocking or advisory. Use when a solution reaches the public or a broad workforce, or after harmful output is reported.
`golden-dataset`	draft	Deliver, Assure, Improve	Builds, maintains and reviews the golden dataset an AI solution is judged against - real questions or tasks from users, expected answers and sources confirmed by knowledge owners, edge, refusal and adversarial cases, versioning and safe handling of test data - with findings classed blocking or advisory. Use when evaluation needs a trusted reference set, when expected answers have drifted, or when a test set looks too friendly.
`groundedness-testing`	draft	Deliver, Assure, Improve	Tests and reviews whether an AI solution's answers are supported by the permitted sources it cites - claim-level support, citation accuracy, behaviour when knowledge is silent, stale or contradictory, and blending with model or web knowledge - classing findings blocking or advisory. Use when a knowledge-grounded agent or retrieval solution approaches release, after knowledge changes, or when users report invented or wrongly cited answers.
`red-team-plan`	draft	Assure, Launch, Improve	Plans and reviews a red-team exercise for an AI solution - objectives, scope, harm and misuse scenarios, a team with varied perspectives, rules of engagement, safe test environments, evidence recording, tester welfare and follow-through to fixes and re-tests - classing plan gaps and findings blocking or advisory. Use when a tier 3 or 4 solution nears release, when it gains new actions or audiences, or after a serious incident.
`regression-evaluation`	draft	Deliver, Assure, Improve	Decides and reviews what must be re-evaluated after a change to a live AI solution - instructions, knowledge, actions, model or platform updates - selecting the regression subset, comparing with the last baseline while allowing for normal output variation, and recommending release, hold or rollback, with findings classed blocking or advisory. Use when a change is ready to deploy or Microsoft announces a model or platform change.

Credit

Created by Simon Foote as part of CopilotForge. Content is licensed under CC BY 4.0; keep the credit in each skill's license and authors fields when you share or adapt it. See NOTICE.md in the repository.