Catalogue › Microsoft Foundry › foundry-safety-evaluation

Foundry safety evaluation

Configures content safety controls for a Microsoft Foundry solution - input and output filtering, prompt attack detection, groundedness and protected material checks, blocklists - and runs the builder's safety evaluations against the solution's harm profile so the release gate receives evidence. Use when a Foundry model or agent is being prepared for release, when filters block legitimate work, or after a harmful output.

Department

Microsoft Foundry

Lifecycle

Deliver, Assure, Improve

Maturity

builder, scaler, ai-enterprise

Risk tiers

1, 2, 3, 4

Platforms

ai-foundry, azure

Editions

starter, scale, enterprise

Reviewers

evaluation-quality, responsible-ai, security-privacy

Status

draft

Copilot Cowork

Microsoft Foundry plugin for Copilot Cowork, or this skill on its own: Download SKILL.md

Licence

CC BY 4.0, by Simon Foote

Purpose

A Foundry solution is responsible for its own safety layer in a way configured Microsoft agents are not: the team chooses filter settings, decides what happens when a filter fires, and must show the solution behaves safely for its audience. This skill sets those controls from a harm profile of the use, runs automated and manual safety evaluations during build, and hands the reviewer gate a clear evidence set. Safety work goes wrong when default filter settings are accepted without anyone understanding them, when filters are loosened to fix one complaint and never revisited, and when safety is tested once at the end on generic prompts rather than on the solution's real risks.

When to use

• A Foundry model deployment or agent is being prepared for its first release and needs safety controls and evidence.
• Users report that legitimate requests are blocked, and someone proposes relaxing the content filters.
• A harmful, offensive or unsafe output has been reported from a Foundry solution.
• The solution will start reading untrusted content, reach a new audience or use a new model.
• A team wants to know which safety evaluations to automate in the build pipeline.

When not to use

• Reviewing at the gate whether the solution's safety evidence is sufficient - that is the reviewer skill evaluation-quality:ai-safety-testing.
• Planning a red-team exercise for a tier 3 or 4 solution - that is the reviewer skill evaluation-quality:red-team-plan.
• Reviewing prompt injection, exfiltration and tool misuse testing for the security gate - that is the reviewer skill security-privacy:ai-security-testing.
• Comparing candidate models on task quality - use ai-foundry:model-evaluation.
• Deciding whether the use is acceptable at all - that is the reviewer skill responsible-ai:prohibited-use-review.

Questions to ask first

1. Who will use the solution, and who could be harmed by its outputs even if they never use it?
2. Which harms matter most for this use: harmful content categories, unsafe or regulated advice, invented facts, exposure of protected material, or manipulation through injected instructions?
3. Will the solution read content the organisation does not control, and can it take actions?
4. What should the user see, and what should be logged, when a filter blocks an input or output?
5. What is the risk tier, and which reviewer gates and thresholds already apply?

Required inputs

• The solution design on the foundry-solution-design template with risk tier, audience and tool list.
• The impact assessment from responsible-ai:ai-impact-assessment where one exists, or at least a list of foreseeable harms.
• The evaluation plan on the evaluation-plan template, with safety thresholds agreed before testing.
• Access to a non-production deployment configured exactly like production.

Where the harm profile or thresholds are missing, mark them [To confirm: ...] and do not declare the solution safe. Testing without an agreed threshold produces numbers nobody can act on.

Decision framework

Practitioner approach

Start from the harm profile, not the filter menu. With the business owner, list who could be harmed and how, using the impact assessment where it exists. A finance summariser and a public-facing advice agent need very different controls, and the profile is what justifies each setting to a reviewer later.

Configure controls in layers. Platform content filtering on inputs and outputs is the first layer; prompt attack detection guards against direct and indirect injection; groundedness and protected material checks address invented claims and reproduced copyrighted content where the solution's outputs warrant them. The system prompt adds refusal behaviour, and tool permissions and confirmation steps contain anything that could still slip through. No single layer is sufficient, so record what each one is for.

Decide what happens when a control fires. Users should get a short, honest message and a route forward, not a raw error or silence. Log the event with the category and the deployment version, but think carefully before logging the full blocked content, which may itself be harmful or personal.

Build a safety evaluation set alongside the quality set. Include the harms in the profile, realistic misuse attempts, injected instructions inside documents the solution reads, sensitive but legitimate requests that must not be blocked, and users in distress. Run automated safety evaluators where the platform provides them, but read a sample of outputs yourself, because automated scores miss context. Record results against thresholds agreed before testing.

Treat over-blocking as a safety finding too. When legitimate work is blocked, users route around the solution into unapproved tools. Before loosening a category, look for a narrower fix: better instructions, a blocklist exception, or a redesigned input. Any loosening is re-evaluated and recorded.

Automate the safety run in the pipeline so every prompt, model or configuration change repeats it, and repeat manual review when the audience or tools change. Assemble the results, configuration and known limits into the safety section of the solution design for the evaluation-quality and responsible-ai gates.

Be clear about limits. Safety evaluation shows how the solution behaved on the cases tried; it cannot prove the absence of harm. Say which harms were not tested and why.

Outputs

• A harm profile and control configuration record in the safety section of the foundry-solution-design template: each control, its setting, why, and the behaviour when it fires.
• A safety evaluation set and results against agreed thresholds, recorded on the evaluation-plan template.
• A log of every filter change with evidence and approver.
• A list of untested harms and residual risks for responsible-ai:ai-risk-acceptance where needed.

Microsoft platform considerations

Microsoft Foundry provides content filtering, prompt attack detection, groundedness and protected material checks and safety evaluators, and their categories, configurability, model coverage and preview status change frequently. Some settings may require approval to change. Check the current source records before stating what a control does or which models and deployment types it covers, and confirm whether third-party models in the catalogue receive the same controls as models sold by Microsoft.

Adoption considerations

Users trust a solution that refuses clearly and offers a way forward, and abandon one that blocks without explanation. Agree the refusal and block messages with the business owner and monitor how often legitimate users hit them.

Knowledge considerations

Grounding content can carry injected instructions or harmful material. Agree with knowledge owners which sources are trusted, and treat external or user-supplied content as untrusted regardless of where it is stored.

Security and privacy considerations

Prompt attack detection is a security control as much as a safety one; keep it on for any solution that reads untrusted content. Logged blocked content may contain personal or harmful data, so set its access and retention with security-privacy:ai-interaction-retention.

Responsible AI considerations

The harm profile must consider people affected by outputs who never use the solution, and groups who may be served worse. Coordinate with responsible-ai:fairness-review where outputs concern people, and never trade a known serious harm for a delivery date.

Evaluation requirements

Safety thresholds are set per harm category before testing, including a ceiling on false blocks of legitimate requests. Every model, prompt, filter or audience change repeats the automated safety run; tier 3 and 4 also repeat manual review and red-team cases.

Value measures

Measure harmful-output incidents in production, false-block rate on legitimate requests and time to fix safety findings, at solution level, owned by the solution owner with the responsible AI lead.

Scale notes

Starter

Use platform defaults with prompt attack detection on, a small safety set built from the harm profile, and a manual review of results before release.

Scale

Share a baseline control configuration and a reusable safety set by audience type, with solution-specific additions, and run the safety evaluation automatically in every pipeline.

Enterprise

Govern control configurations centrally with recorded exceptions, track safety incidents across all Foundry solutions, and feed common failure patterns back into the shared baseline.

Hand-offs

• To evaluation-quality:ai-safety-testing for the gate review of the evidence.
• To evaluation-quality:red-team-plan for tier 3 and 4 solutions.
• To ai-foundry:foundry-prompt-engineering when refusals or tone need prompt changes.
• To ai-foundry:foundry-observability so safety events are traced and alerted in production.
• To ai-operations:ai-incident-management when a harmful output reaches users.

Reviewer gates

From risk tier 2, evaluation-quality checks safety evidence against thresholds set in advance, responsible-ai checks the harm profile covers affected people, and security-privacy checks prompt attack controls. A filter loosened without re-evaluation is a blocking finding at any tier.

Failure modes

• Default filters are accepted without review, and a public-facing solution gives unsafe advice in a category the defaults allow.
• A filter is loosened to stop one complaint, never re-tested, and harmful content passes for months.
• Safety is tested only with generic attack prompts, and injected instructions in the solution's own documents go untested.
• Over-blocking drives staff back to unapproved tools, and the solution is judged safe because nobody uses it.

Sources

references/sources.md in this skill lists the source records behind this skill, what each is authoritative for, how volatile it is and what you may do with it. Product availability, licensing, limits and preview status change often, so check the current source before stating them, and cite what you used.

Never

• Declare a solution safe without a harm profile and thresholds agreed before testing.
• Switch off prompt attack detection on a solution that reads untrusted content.
• Loosen a content filter without evidence, re-evaluation and a recorded approver.
• Claim that safety evaluation proves the absence of harm.