Catalogue › AI Operations › ai-review-calendar

AI review calendar

Maintains a calendar of scheduled reviews for every live agent and AI solution, with frequency set by risk tier, a standard agenda covering ownership, quality, usage, feedback, incidents, knowledge, risk tier and cost, and a recorded outcome to continue, improve, re-tier, escalate or retire. Use when live solutions have no review dates, when reviews keep slipping, or when nobody can say which solutions are still needed.

Department

AI Operations

Lifecycle

Measure, Scale, Improve

Maturity

adopter, builder, scaler, ai-enterprise

Risk tiers

1, 2, 3, 4

Platforms

m365-agents, copilot-studio, copilot-cowork, ai-foundry, power-platform, platform-neutral

Editions

starter, scale, enterprise

Reviewers

none

Status

draft

Copilot Cowork

AI Operations plugin for Copilot Cowork, or this skill on its own: Download SKILL.md

Licence

CC BY 4.0, by Simon Foote

Purpose

Every live AI solution should be looked at on a schedule, not only when something breaks. Owners change, knowledge ages, platforms update, usage shifts and risk tiers drift as solutions acquire new audiences or actions. This skill keeps a calendar of scheduled reviews for every live solution, sets their frequency by risk tier, gives each review a standard agenda drawing on operational evidence, and records an outcome. Review calendars usually fail because review dates are set at release and then forgotten, because reviews become a tick-box with no evidence, or because nobody has the authority to act on the outcome.

When to use

• Live solutions have no scheduled review dates.
• Review dates exist but keep slipping, and nobody follows up.
• Nobody can say which live solutions are still needed, used or owned.
• A new release gate requires a review date and the organisation needs a consistent approach.
• An audit or management system review asks how live AI solutions are overseen.

When not to use

• Deciding to scale, iterate, pivot or stop after a pilot - use opportunity-management:scale-or-stop.
• Comparing realised benefits with the business case - use value-realisation:post-implementation-review.
• Setting the CoE's own meeting rhythm and terms of reference - use ai-centre-of-excellence:coe-cadence.
• Maintaining the inventory of agents, flows and solutions - use platform-governance:ai-solution-inventory.
• Managing one agent through its whole lifecycle - use m365-agents:agent-lifecycle or copilot-studio:studio-agent-lifecycle.

Questions to ask first

1. Which solutions are live, and where is the authoritative list?
2. Who owns each solution, and who has authority to decide its future?
3. What evidence is available for review - monitoring, feedback, incidents, evaluation, knowledge freshness, cost?
4. What happens when a review is overdue or an owner does not respond?
5. Which forum receives review outcomes that need a decision beyond the owner?

Required inputs

• The solution inventory with owners, risk tiers and release dates, from platform-governance:ai-solution-inventory or the CoE register.
• Monitoring trends, feedback themes, incident records and refresh logs for each solution.
• The release evidence and value measures recorded at release.
• The organisation's rule for what happens to overdue or ownerless solutions.

Where the inventory is incomplete, the calendar starts with what is known and a discovery task for the rest; never treat an unlisted solution as reviewed.

Decision framework

Set frequency by tier and adjust for events. The frequencies are a CopilotForge starting convention; set your own and record them.

Standard agenda for every review:

Outcomes: continue; improve with actions; re-tier and re-review through the relevant gates; escalate for a scale-or-stop decision; retire.

Practitioner approach

Build the calendar from the inventory. Every live solution gets a next review date set by its tier, an owner and a forum. Put review dates where owners will see them, in their own calendars as well as the register. A calendar only the operations team can see is a list of missed meetings.

Prepare the evidence pack, not the owner's memory. Before each review, gather monitoring trends, a quality sample, feedback themes, incidents and refresh logs. Keep it short: one page per solution with exceptions highlighted. The aim is a fifteen-minute conversation for a healthy tier 1 or 2 solution, and a deeper one only where evidence demands it.

Run the agenda in order and record each answer. Ownership comes first, because a solution without an active owner cannot be improved or defended. Purpose comes second: many solutions drift from their original scope, and a solution used for something it was never evaluated for is a risk tier question.

Re-tier honestly. If an agent built for one team now serves a department, or has gained an action, its tier may have risen. Send it back through opportunity-management:risk-classification and the gates of the new tier before the next release of anything.

Record one outcome per review, with actions, owners and dates. "Continue" is a valid outcome, but it needs evidence. "Retire" is equally valid and should be welcomed, not avoided; unused solutions carry cost, risk and maintenance for no value.

Handle overdue reviews firmly. Agree a rule with the CoE: a reminder, then escalation to the business owner, then a restriction such as narrowing sharing, and finally retirement for solutions that remain unreviewed and ownerless. Apply the rule consistently, and tell owners in advance.

Feed the portfolio. Summarise review outcomes for the CoE on its cadence: solutions reviewed on time, outcomes, re-tiered solutions and retirements. Patterns across reviews, such as repeated knowledge expiry, inform standards and training.

Outputs

• A review calendar for all live solutions with tier, frequency, next date, owner and forum, recorded in each support-and-monitoring-plan.
• A one-page review pack template and completed packs per review.
• A review record per solution with outcome, actions, owners and dates.
• A portfolio summary for the CoE cadence.

Microsoft platform considerations

Inventory and admin tooling, including the CoE Starter Kit and Managed Environments features, can help list solutions, find inactive ones and identify owners; their capabilities change, so confirm them in ms-coe-starter-kit and ms-managed-environments. Environment zoning described in ms-copilot-studio-zoned-governance can guide which solutions need closer review.

Adoption considerations

Review outcomes are a source of good news as well as problems. Share examples of solutions that improved after review or were retired cleanly, so owners see reviews as useful rather than bureaucratic.

Knowledge considerations

Knowledge freshness is a standing agenda item. Align review dates with knowledge owners' cycles where possible, so one conversation covers both.

Security and privacy considerations

Reviews check that sharing scope and connections still match what was approved. Any widening since release goes back to the security-privacy reviewers before the solution continues.

Responsible AI considerations

For solutions affecting people, reviews include whether oversight arrangements still work and whether any complaints or appeals arose, with input from the responsible-ai reviewers at tier 3 and 4.

Evaluation requirements

Each review includes a quality sample scored against the release rubric. Results below threshold trigger evaluation-quality:regression-evaluation before the outcome is "continue".

Value measures

Track the share of live solutions reviewed on time, outcomes by type, solutions re-tiered and solutions retired, and time from overdue to resolution. The AI Operations Lead owns these measures; report by solution and portfolio, never by individual owner performance.

Scale notes

Starter

A simple list of live agents with owners and next review dates, reviewed by the owner with whoever leads AI, using the standard agenda on one page.

Scale

A maintained calendar linked to the solution inventory, evidence packs prepared by operations, reviews in the CoE cadence for tier 2 and above, and a consistent overdue rule.

Enterprise

Federated reviews by business units for tier 1 and 2 under the same agenda, central reviews for tier 3 and 4 with reviewer input, and portfolio reporting to the CoE and risk committees.

Hand-offs

• To opportunity-management:scale-or-stop when a review escalates a scale or stop decision.
• To opportunity-management:risk-classification when a solution needs re-tiering.
• To ai-operations:ai-solution-retirement when the outcome is retire.
• To value-realisation:post-implementation-review when value evidence needs a fuller review.

Reviewer gates

Not applicable as a gate in itself; scheduled reviews are an operating control. Where a review re-tiers a solution or finds widened sharing, the security-privacy, evaluation-quality and responsible-ai gates for the new tier apply before it continues.

Failure modes

• Review dates are set at release and never appear in anyone's calendar.
• Reviews rely on the owner's impressions with no evidence, and every solution is "continue".
• A solution's audience grows without re-tiering, and it runs at tier 3 with tier 1 controls.
• Overdue reviews have no consequence, so ownerless solutions accumulate.

Sources

references/sources.md in this skill lists the source records behind this skill, what each is authoritative for, how volatile it is and what you may do with it. Product availability, licensing, limits and preview status change often, so check the current source before stating them, and cite what you used.

Never

• Leave a live solution without a next review date and an owner.
• Record "continue" without evidence of quality, ownership and purpose.
• Use reviews to assess or rank individual owners or makers.
• Let a solution run at a higher effective tier than the gates it passed.