Catalogue › AI Operations › ai-incident-management

AI incident management

Runs incident management for AI failures and harms in live solutions - recognising AI incident types, setting severity, containing fast, preserving evidence, communicating, involving security, privacy and legal roles, and post-incident review before return to service. Use when an agent gives harmful, exposing or widely wrong outputs, takes a wrong action, or when AI incident procedures are being prepared.

Department

AI Operations

Lifecycle

Launch, Measure, Improve

Maturity

adopter, builder, scaler, ai-enterprise

Risk tiers

1, 2, 3, 4

Platforms

m365-copilot, m365-agents, copilot-studio, copilot-cowork, ai-foundry, power-platform

Editions

starter, scale, enterprise

Reviewers

security-privacy, responsible-ai

Status

draft

Copilot Cowork

AI Operations plugin for Copilot Cowork, or this skill on its own: Download SKILL.md

Licence

CC BY 4.0, by Simon Foote

Purpose

AI incidents rarely look like outages. The solution is up and answering - but it is telling staff the wrong leave entitlement, summarising a confidential file for the wrong audience, sending customers an unapproved commitment or responding badly to someone in distress. This skill gives operations a way to recognise AI incidents, set their severity, contain them quickly with the levers each route offers, preserve evidence, bring in the right specialists and learn from them before returning the solution to service. Incident handling goes wrong when a harmful output is logged as an ordinary fault, when nobody knows how to switch a solution off, or when the solution goes back live with no proof the cause was fixed.

When to use

• A live solution produces harmful, offensive, exposing or widely wrong outputs, or takes a wrong action.
• A monitoring trigger or user report suggests one of those may be happening.
• The organisation is preparing incident procedures for AI solutions before go-live.
• A post-incident review is needed after an AI-related event.

When not to use

• Reviewing whether the organisation is ready to handle AI security and privacy incidents - use security-privacy:ai-incident-readiness.
• Routine wrong answers handled through normal support - use ai-operations:ai-support-model.
• Planned fallback when an AI service is unavailable - use ai-operations:ai-continuity.
• Routes for affected people to challenge an outcome - use responsible-ai:escalation-and-appeal.
• Deciding whether to retire a solution after repeated incidents - use ai-operations:ai-solution-retirement after the review.

Questions to ask first

1. What happened, to whom, and is it still happening?
2. Could restricted or personal data have been exposed, or could people have been harmed or treated unfairly?
3. Did the solution act in another system or communicate outside the organisation?
4. What is the fastest safe containment - restrict sharing, disable an action, remove a knowledge source or switch the solution off?
5. Who must be told now - solution owner, security, DPO, legal, HR, communications, the accountable executive?

Required inputs

• The solution's runbook with containment steps and owners, from ai-operations:ai-support-model.
• The organisation's existing incident, security incident and personal data breach procedures.
• Contacts for security, the DPO, legal and HR, and the accountable executive for the solution.
• Access to logs, transcripts or monitoring data that are retained for the solution.

Where a solution has no documented way to switch it off or restrict it, record that as a gap and find the route with the platform team immediately; containment comes before diagnosis.

Decision framework

Classify the incident type and set severity by the highest applicable factor.

Containment levers, fastest first: restrict sharing to a smaller group; disable a specific action or connector; remove or restrict a knowledge source; show a warning to users; switch the solution off. Choose the least disruptive lever that stops the harm.

Practitioner approach

Contain first. When a credible report arrives, the incident lead's first question is whether harm is continuing. If it is, pull the fastest lever that stops it, even before the cause is known. Switching off a well-used agent for an afternoon is a small cost against continued exposure or harm. Record what was done and when.

Preserve evidence before it disappears. Capture the user's report, screenshots, the question and answer, the time, the solution configuration version and any logs or transcripts that are retained. Retention varies by route and configuration, so check what exists and secure it quickly within the organisation's policy.

Bring in the right people by type. Data exposure goes to the security and privacy teams under the organisation's breach procedure; the DPO decides whether any notification is needed. Harm to people goes to HR or the relevant owner and to the responsible AI lead. Wrong external communications go to communications and legal. The incident lead coordinates; the specialists decide in their areas.

Diagnose the cause with the solution owner and maker: knowledge (a superseded or overshared source), instructions, configuration (a setting changed, web grounding enabled), an action or connector, a platform or model change, or misuse. Many incidents have more than one cause. Check whether other solutions share the same cause, such as the same overshared library.

Communicate plainly. Tell affected users what happened, what they should do (for example, disregard an earlier answer) and when the solution will return. Do not speculate in communications, and agree wording with legal or communications for critical incidents.

Return to service only with evidence. Fix the cause, re-run the affected items and the regression subset through evaluation-quality:regression-evaluation, add the incident's prompt pattern to the golden dataset, and get the business owner's agreement. For critical incidents, the relevant reviewers confirm the fix.

Review without blame. Within a short period, hold a post-incident review covering timeline, causes, what worked, what did not and actions with owners. Focus on systems and decisions, never on individual fault. Share lessons with other solution owners and feed patterns into the review calendar.

Outputs

• An incident record: type, severity, timeline, containment, evidence, specialists engaged, communications, cause, fix and return-to-service evidence.
• A post-incident review with actions, owners and dates.
• New regression items for evaluation-quality:golden-dataset.
• An AI incident procedure and severity matrix for the organisation, with containment levers per route recorded in each solution's support-and-monitoring-plan.

Microsoft platform considerations

The levers for containment differ by route: sharing and availability settings for agents, environment and data policy controls in Power Platform, and deployment controls in Microsoft Foundry. Confirm current controls in ms-copilot-studio-security-governance and ms-managed-environments, and confirm what interaction data is retained and how it can be accessed in ms-copilot-privacy. Practise the switch-off route before go-live.

Adoption considerations

How an incident is handled shapes trust more than the incident itself. Prompt, plain communication and a visible fix keep users; silence drives them to unapproved tools.

Knowledge considerations

Many incidents trace to knowledge: overshared libraries, superseded policies, inappropriate content in grounding. Fix the source, and check whether other solutions use it.

Security and privacy considerations

Data exposure incidents follow the organisation's security incident and personal data breach procedures, led by the security team and the DPO, who decide on any notification. Readiness for these is reviewed through security-privacy:ai-incident-readiness.

Responsible AI considerations

Incidents that harm people or treat them unfairly are reported to the responsible-ai reviewers and may require contacting affected people through responsible-ai:escalation-and-appeal. Repeated harm may mean the use should stop.

Evaluation requirements

Return to service requires re-running the affected items and the regression subset with results at or above threshold, and adding the incident pattern to the golden dataset.

Value measures

Track incidents by type and severity per solution, time to containment, time to return to service, and the share of post-incident actions closed on time. The AI Operations Lead owns these measures; report by solution, never by individual maker or user.

Scale notes

Starter

A one-page AI incident procedure: how to report, who decides, how to switch each agent off, and who to call for data exposure. Practise the switch-off once before go-live.

Scale

AI incident types and severities added to the service management process, containment steps in each runbook, and post-incident reviews shared across solution owners.

Enterprise

AI incidents integrated with security and enterprise incident management, rehearsed scenarios for tier 3 and 4 solutions, portfolio-level trend analysis and lessons fed into standards and reviewer criteria.

Hand-offs

• To evaluation-quality:regression-evaluation before return to service.
• To security-privacy:ai-incident-readiness with lessons that change readiness.
• To ai-operations:ai-review-calendar with incident patterns for scheduled reviews.
• To ai-operations:ai-solution-retirement when the review recommends retirement.

Reviewer gates

From risk tier 2, security-privacy confirms the fix for exposure incidents and responsible-ai confirms the fix for harm-to-people incidents before return to service. Not applicable as a release gate otherwise; the incident procedure itself is reviewed through security-privacy:ai-incident-readiness.

Failure modes

• A harmful output is logged as an ordinary fault and closed with a note to the maker.
• Nobody knows how to switch the solution off, and harm continues while people search for the setting.
• Evidence is lost because transcripts were not secured before retention removed them.
• The solution returns to service with an instruction patch and no regression run.

Sources

references/sources.md in this skill lists the source records behind this skill, what each is authoritative for, how volatile it is and what you may do with it. Product availability, licensing, limits and preview status change often, so check the current source before stating them, and cite what you used.

Never

• Delay containment of continuing harm to finish diagnosis.
• Decide on personal data breach notification without the DPO.
• Return a solution to service without re-running affected items and the regression subset.
• Run a post-incident review that assigns blame to individuals.