Catalogue › AI Engineering › ai-engineering-standards

AI engineering standards

Sets and maintains engineering standards for custom AI work - minimum practices by risk tier for repositories, identity, tools, testing, evaluation, pipelines, telemetry, performance, resilience and ownership - with a checklist, exception route and review cycle. Use when pro-code AI work starts, when teams build AI solutions inconsistently, or when standards need refreshing after platform change.

Department

AI Engineering

Lifecycle

Mobilise, Deliver, Assure, Scale, Improve

Maturity

builder, scaler, ai-enterprise

Risk tiers

1, 2, 3, 4

Platforms

ai-foundry, azure, github, copilot-code, entra

Editions

starter, scale, enterprise

Reviewers

design-authority, security-privacy, evaluation-quality

Status

draft

Copilot Cowork

AI Engineering plugin for Copilot Cowork, or this skill on its own: Download SKILL.md

Licence

CC BY 4.0, by Simon Foote

Purpose

When each team builds custom AI solutions its own way, reviewers face a new design every time, operations inherit solutions they cannot support, and the same mistakes (keys in code, prompts edited in production, no evaluation gate) recur across the estate. This skill sets a short, enforceable set of engineering standards for pro-code AI work, scaled by risk tier, with a checklist teams use during build, an exception route for justified departures and a review cycle that keeps the standards current as Microsoft's platforms change. It goes wrong when standards are long documents nobody reads, when every rule applies to every prototype, and when standards are written once and never revisited.

When to use

• The organisation is starting pro-code AI work on Microsoft Foundry, Azure or with Copilot Code and needs common standards.
• Teams build AI solutions inconsistently, and reviews keep finding the same issues.
• The design authority needs a reference to review conformance against.
• Platform changes (new deployment options, evaluation features, identity capabilities) make existing standards stale.
• A partner or supplier is building AI solutions and needs to know what the organisation expects.

When not to use

• Reviewing a solution's conformance against the standards - use design-authority:architecture-conformance.
• Maintaining reference architectures for common AI patterns - use ai-architecture:ai-reference-architecture.
• Code review standards specific to AI-generated code - use copilot-code:generated-code-review.
• Build patterns for Power Automate flows - use automation:flow-design-patterns.
• Setting release policy and approvals by risk tier - use platform-governance:ai-release-policy.

Questions to ask first

1. Which kinds of custom AI work does the organisation do or plan (Foundry agents, retrieval solutions, AI APIs, AI-built applications), and how many teams do it?
2. What engineering standards already exist for ordinary software, and which can AI work simply inherit?
3. Which recurring problems have reviews, incidents or audits found in AI solutions so far?
4. Who will own the standards, approve exceptions and review them?
5. How will teams be supported to meet the standards (templates, shared components, clinics)?

Required inputs

• Existing software engineering, security and change standards, so AI standards add to rather than duplicate them.
• The risk tier model from opportunity-management:risk-classification.
• Findings from reviews, incidents and post-implementation reviews of AI solutions, where any exist.
• The organisation's platforms and tooling for source control, pipelines, identity and monitoring.

If no existing engineering standards exist, start with the AI-specific minimum in the decision framework and record the wider gap for the engineering lead. Never present draft standards as mandatory before the owner has approved them.

Decision framework

Standards are stated as testable requirements, each with the tier from which it is mandatory and the evidence that shows it is met. This is the core set; the engineering-standards-checklist template carries the full checklist.

Prototypes in sandbox environments with no real users or sensitive data may follow a lighter set, recorded as such, and must meet the full set before any wider use.

Practitioner approach

Start from what goes wrong. Gather findings from reviews, incidents and post-implementation reviews, and the recurring questions reviewers ask. Standards that prevent problems the organisation has actually had are easier to justify and adopt than a generic catalogue.

Inherit before inventing. Most good software practice (version control, code review, dependency scanning, change control) applies to AI solutions unchanged. Point to existing standards and add only what AI work needs: prompts and model configuration as versioned assets, evaluation as a gate, model version lifecycle, content capture in telemetry, tool safety and model-specific failure handling.

Write each standard as a testable requirement with evidence. "Solutions should be well tested" cannot be checked; "failure paths for model timeout, content filter block and malformed output are covered by automated tests" can. For each standard, name the evidence a reviewer looks for and the risk tier from which it is mandatory. Keep the core set short enough to fit on a page; detail lives in the linked skills and templates.

Scale by tier. A tier 1 internal tool and a tier 4 customer-facing service should not carry the same burden. The tier column makes proportionality explicit and stops prototypes being smothered or production systems being under-governed.

Make conformance easy. Back every standard with something that helps meet it: a template repository, pipeline templates, shared evaluators, identity starter code, telemetry libraries. A standard without an easy path generates exceptions and resentment. Run clinics where teams can ask how to meet a standard before a review finds they have not.

Provide an exception route. Some departures are justified. An exception states the standard, the reason, the compensating control, the risk accepted, who approved it and when it will be reviewed. Exceptions are recorded against the solution and reviewed; a pattern of exceptions to one standard means the standard or its support needs changing.

Keep it current. Microsoft's AI platforms change quickly. Review the standards on a fixed cycle and when a significant platform change lands, checking the source records rather than memory. Publish a short change note with each revision so teams know what changed and why. Retire standards that no longer prevent anything.

Outputs

• The AI engineering standards: a short core set by area, each with requirement, evidence and mandatory tier.
• The engineering-standards-checklist template, used by teams during build and by reviewers at the gate.
• An exception register: standard, solution, reason, compensating control, approver, review date.
• A revision history with change notes, and the date of the next scheduled review.

Microsoft platform considerations

Standards should describe outcomes (keyless access, versioned prompts, gated evaluation) rather than depend on a specific product feature, so they survive renames and changes. Where a standard relies on a Microsoft capability, such as Foundry evaluation, managed identities or a deployment type, link to the source record in references/sources.md and recheck it at each review.

Adoption considerations

Engineers adopt standards they helped shape and that save them effort. Draft with practising engineers, pilot on one or two solutions, and lead with templates and shared components rather than enforcement. Report conformance by solution and team, never by individual engineer.

Knowledge considerations

The standards and checklist are knowledge assets in their own right: keep them in one maintained location, owned by the AI engineering lead, and point skills, templates and reviewers to that location rather than copies.

Security and privacy considerations

Identity, secrets, tool authorisation, adversarial testing and telemetry content capture are the security and privacy core of the standards. Agree these sections with the security and privacy lead so the standards and the security-privacy reviews ask the same questions.

Responsible AI considerations

Standards should require that safety evaluation, human confirmation for consequential actions and honest failure behaviour are engineered in, not added later. Align the relevant standards with the responsible-ai department's review expectations.

Evaluation requirements

Evaluation as a gate is itself a standard. Align thresholds, harness requirements and evidence with the evaluation-quality department so teams meet one set of expectations.

Value measures

Track conformance by standard across solutions, recurring review findings, open exceptions past review date and incidents traceable to a missing standard. The AI engineering lead owns these and reports them to the design authority.

Scale notes

Starter

A one-page set of the most important standards for tier 1 and 2 work, a template repository and a short checklist at release. The AI engineering lead approves exceptions.

Scale

The full core set by tier with the checklist, shared components behind each standard, exception register and a fixed review cycle aligned with design authority reviews.

Enterprise

Standards governed through the design authority with federated contribution from engineering teams, automated conformance checks in pipelines where possible, and estate-wide conformance reporting.

Hand-offs

• To design-authority:architecture-conformance to review solutions against the standards.
• To ai-engineering:ai-solution-repository and ai-engineering:ai-ci-cd for templates that implement them.
• To platform-governance:ai-release-policy to align gates with release policy.
• To ai-architecture:ai-reference-architecture where standards and reference architectures need to agree.

Reviewer gates

From risk tier 2, design-authority reviews solutions against these standards, with security-privacy and evaluation-quality reviewing their sections. The standards themselves are approved by the design authority, and an unapproved exception to a mandatory standard at the solution's tier is a blocking finding.

Failure modes

• Standards are a long document nobody reads, and reviews keep finding the same issues.
• Every standard applies to every prototype, so teams build outside governance to avoid the burden.
• Standards are written once, platforms move on, and the rules end up demanding obsolete practices.
• Exceptions are granted informally, never reviewed, and quietly become the norm.

Sources

references/sources.md in this skill lists the source records behind this skill, what each is authoritative for, how volatile it is and what you may do with it. Product availability, licensing, limits and preview status change often, so check the current source before stating them, and cite what you used.

Never

• Publish a standard that cannot be tested with named evidence.
• Apply production standards to sandbox prototypes, or sandbox leniency to production solutions.
• Grant an exception without a reason, a compensating control, an approver and a review date.
• Use conformance data to rank individual engineers.