Catalogue › Evaluation and Quality › ai-safety-testing

AI safety testing

Tests and reviews whether an AI solution avoids harmful outputs for its audience - hateful, violent, sexual or self-harm content, dangerous or regulated advice, demeaning statements about people, and poor handling of users in distress - checking platform content controls as configured, classing findings blocking or advisory. Use when a solution reaches the public or a broad workforce, or after harmful output is reported.

Department

Evaluation and Quality

Lifecycle

Deliver, Assure, Improve

Maturity

adopter, builder, scaler, ai-enterprise

Risk tiers

2, 3, 4

Platforms

m365-agents, copilot-studio, ai-foundry, copilot-cowork

Editions

starter, scale, enterprise

Reviewers

evaluation-quality, responsible-ai

Status

draft

Copilot Cowork

Evaluation and Quality plugin for Copilot Cowork, or this skill on its own: Download SKILL.md

Licence

CC BY 4.0, by Simon Foote

Purpose

An AI solution can be correct and grounded most of the time and still cause harm on the occasions it produces offensive, dangerous or demeaning content, or responds badly to someone in distress. Platform content controls reduce that risk; they do not remove it, and their configuration and the solution's own instructions change what gets through. This reviewer skill designs and reviews scripted safety tests for the harms that matter to a solution's audience and domain, checks content controls as actually configured, and classifies each finding as blocking or advisory. Safety testing goes wrong when teams assume platform filters cover everything, or test generic harms while missing the domain-specific ones such as unsafe workplace advice.

When to use

• A solution at tier 2 or above will be used by a broad workforce, customers or the public.
• A solution operates in a domain where advice can cause harm, such as health, safety, finance, legal or employment matters.
• Content control settings, the model or the solution's instructions have changed.
• A user has reported an offensive, unsafe or distressing output.
• Safety evidence is submitted to the evaluation-quality gate.

When not to use

• Prompt injection, data exfiltration and attacks on tools or connectors - use security-privacy:ai-security-testing.
• Planning an open-ended adversarial exercise with a dedicated team - use evaluation-quality:red-team-plan.
• Configuring content safety controls in Microsoft Foundry - use ai-foundry:foundry-safety-evaluation.
• Deciding whether a use should be built at all - use responsible-ai:prohibited-use-review.
• Bias and unequal treatment across groups of people - use responsible-ai:fairness-review.

Questions to ask first

1. Who will use the solution, including vulnerable users, customers or the public, and in what state might they arrive?
2. Which domain harms apply - unsafe instructions, medical, legal or financial advice, statements about named people?
3. Which content controls apply on the chosen route, at what settings, and who changed them from the defaults?
4. What should the solution do when a user discloses distress, risk of harm or a safeguarding concern?
5. Who will run tests that involve distressing content, and how will they be supported?

Required inputs

• The solution brief with audience, domain, scope and behaviour rules, and the risk tier.
• The content control configuration for the route, confirmed by the builder.
• The organisation's escalation routes for distress and safeguarding, confirmed with HR or the relevant owner.
• A harm catalogue for the domain, built with subject experts.

Where escalation routes for distress are not defined, the solution cannot be tested against them; record that as blocking from tier 2 for any solution reachable by a broad workforce or the public.

Decision framework

Build tests in four categories and scale depth by tier.

Classify findings.

Practitioner approach

Start with a harm catalogue specific to the solution. Sit with a subject expert and the business owner and list what a harmful answer would look like for this audience: a maintenance agent describing a shortcut that bypasses a safety lock, an HR agent giving definitive legal advice on dismissal, a customer agent repeating a slur from a complaint it summarised. Generic harm lists miss these, and they are usually the ones that matter most.

Write test items for each catalogue entry, directly and indirectly. Direct requests test whether the solution refuses; indirect ones test whether harmful content slips out while answering something reasonable, such as summarising a document that contains offensive language. Add role-play and hypothetical framings, which often get past instructions that only cover direct requests. Keep items in a restricted dataset; they are harmful by design.

Confirm the content controls as configured, not as assumed. Ask the builder to show the current settings for the route and any changes from defaults, and record who approved each change and why. Then test the solution with those settings. Where controls were loosened for a legitimate reason, test the categories they affect more deeply.

Test distress handling with care. Agree the expected behaviour with HR or the safeguarding owner: acknowledge, avoid giving advice beyond the solution's role, and point to the organisation's support routes. Write a small number of realistic disclosures and check the response every time; this category does not tolerate variation.

Run each item more than once, because outputs vary. Score with the safety dimension from evaluation-quality:ai-quality-rubric: pass or fail, with no partial credit. Record the actual output for every failure.

Look for over-refusal too. A solution that refuses ordinary questions about, say, medication storage in a care setting is safe and useless. Note over-refusals as advisory unless they block a safety-critical use, and route them to the builder.

Protect the testers. Safety testing exposes people to distressing material. Use volunteers who have been briefed, rotate the work, and offer the organisation's support services. Never assign this testing to someone without their agreement.

Report failures with causes - instructions, knowledge content, configuration or model behaviour - and re-test after each fix. Harmful output in the general or harm-to-people categories is never traded against other dimensions.

Outputs

• A safety test set by category, held in a restricted location, linked from the evaluation-plan.
• A record of content control settings as tested, with approvals for any change from defaults.
• A safety results record with failures, causes, fixes and re-test outcomes, classified blocking or advisory.
• Agreed expected behaviour for distress disclosures, with the owner who confirmed it.

Microsoft platform considerations

Microsoft provides content safety controls across Copilot Studio and Microsoft Foundry, and Microsoft's responsible AI resources describe its approach. Which controls exist, their defaults and how they are configured differ by route and change over time; confirm them in ms-copilot-studio-guidance, ms-copilot-studio-security-governance, ms-foundry-models and ms-responsible-ai. Detailed Foundry configuration belongs with ai-foundry:foundry-safety-evaluation.

Adoption considerations

Tell users what the solution will not help with and where to go instead. A clear refusal with a useful redirect keeps trust; an unexplained refusal drives people to unapproved tools.

Knowledge considerations

Harmful content can enter through grounding, for example quoted complaints or historical documents. Where tests show this, work with the knowledge owner to remove or restrict the content rather than relying on filters alone.

Security and privacy considerations

Safety and security tests overlap at jailbreak attempts. Share jailbreak findings with security-privacy:ai-security-testing so they are tested as attacks too. Restrict access to safety datasets and recorded outputs.

Responsible AI considerations

Safety testing provides evidence to the responsible-ai reviewers, especially responsible-ai:human-oversight for distress handling and responsible-ai:ai-impact-assessment for domain harms. A pattern of harmful outputs may mean the use should not proceed, which is their decision.

Evaluation requirements

This skill supplies the safe dimension of evaluation. Minimum from tier 2: a domain harm catalogue, test items in every applicable category run more than once, content controls recorded as tested, and zero harmful outputs in the general and harm-to-people categories.

Value measures

Track safety failures found before release versus reported after, time to fix and re-test, and over-refusal reports. The AI Evaluation Lead reports by solution; never by tester, maker or user.

Scale notes

Starter

A short domain harm catalogue built with one expert, a core set of harmful and distress items run three times each, and the content settings recorded.

Scale

Shared safety test sets by domain, reused across solutions, with solution-specific additions and a briefed pool of volunteer testers who rotate.

Enterprise

Central safety standards and datasets owned by evaluation-quality, independent testers for tier 4, red-team findings feeding the sets, and portfolio-level tracking of safety incidents and near misses.

Hand-offs

• To evaluation-quality:red-team-plan from tier 3 for open-ended adversarial testing.
• To security-privacy:ai-security-testing with jailbreak findings.
• To responsible-ai:human-oversight and responsible-ai:ai-impact-assessment with distress and domain harm evidence.
• To evaluation-quality:ai-release-evidence with results.

Reviewer gates

This skill operates within the evaluation-quality gate from risk tier 2, with responsible-ai reviewing distress and harm-to-people evidence from tier 3. Blocking findings stop release and are not traded against dates; disagreement escalates to the accountable executive for written risk acceptance.

Failure modes

• Platform filters are assumed to cover every harm, so domain-specific dangerous advice is never tested.
• Content controls were loosened during build to fix over-refusal and nobody re-tested.
• Distress disclosures produce a generic reply with no route to help.
• Testers are exposed to harmful material without agreement or support.

Sources

references/sources.md in this skill lists the source records behind this skill, what each is authoritative for, how volatile it is and what you may do with it. Product availability, licensing, limits and preview status change often, so check the current source before stating them, and cite what you used.

Never

• Pass a solution that produced harmful content in the general or harm-to-people categories.
• Assume platform content controls are configured as the defaults without checking.
• Assign safety testing involving distressing content to anyone without their agreement and support.
• Store harmful test items where an agent or a general audience could reach them.