Catalogue › AI Engineering › ai-performance-engineering

AI performance engineering

Engineers the latency and throughput of custom AI solutions against agreed targets - latency budgets per step, token budgets, streaming, caching, parallel calls, right-sized models, quota and capacity planning, and load testing - while protecting quality and cost. Use when an AI solution is too slow, when usage is about to grow, when throttling appears under load, or before a tier 3 solution goes to production.

Department

AI Engineering

Lifecycle

Deliver, Assure, Scale, Improve

Maturity

builder, scaler, ai-enterprise

Risk tiers

1, 2, 3, 4

Platforms

ai-foundry, azure

Editions

scale, enterprise

Reviewers

design-authority

Status

draft

Copilot Cowork

AI Engineering plugin for Copilot Cowork, or this skill on its own: Download SKILL.md

Licence

CC BY 4.0, by Simon Foote

Purpose

AI solutions are slow in ways ordinary applications are not: each model call takes seconds, agents chain several calls with retrieval and tools, and capacity is limited by quotas and tokens rather than servers. Users abandon a helpful assistant that takes too long, and a solution that works for a pilot group can throttle when a department arrives. This skill engineers latency and throughput against agreed targets by budgeting time and tokens per step, cutting unnecessary calls, using streaming and caching, choosing appropriately sized models and planning capacity, without quietly trading away quality or blowing the budget. It goes wrong when teams tune without measuring, and when a faster, smaller model is swapped in without re-running evaluation.

When to use

• Users report that an AI solution is too slow, or abandon requests before they complete.
• Usage is expected to grow (a wider rollout, a new channel, a seasonal peak).
• Throttling or quota errors appear under load.
• A tier 3 or 4 solution needs performance evidence before production.
• Cost per request is rising and performance work could reduce tokens and calls.

When not to use

• Setting the performance targets and other non-functional requirements - use ai-architecture:ai-non-functional-requirements.
• Managing model and platform cost as a FinOps discipline - use ai-foundry:foundry-cost-management.
• Choosing deployment types, regions and topology - use ai-foundry:foundry-deployment-architecture.
• Retries, fallbacks and graceful degradation when dependencies fail - use ai-engineering:ai-resilience.
• Comparing models on quality for a task - use ai-foundry:model-evaluation.

Questions to ask first

1. What latency do users need, measured how (time to first token, time to complete answer), and at what percentile?
2. What peak load is expected (concurrent users, requests per minute, tokens per minute), and when?
3. How many model calls, retrieval queries and tool calls does a typical request make, and which are sequential?
4. What quota and capacity does the solution have, in which deployments and regions?
5. What quality thresholds must be maintained, and what is the cost budget per request or per month?

Required inputs

• Agreed performance targets from ai-architecture:ai-non-functional-requirements, or from the business owner.
• Telemetry with per-step latency and tokens, from ai-engineering:ai-telemetry.
• Current deployments, quotas and capacity, and expected load.
• The evaluation harness and thresholds, to check quality after each change.

If no targets have been agreed, measure the current state, propose targets to the business owner and record them before tuning. If per-step telemetry is missing, instrument first; tuning without it is guesswork.

Decision framework

Work through the levers in order; earlier levers are usually cheaper and safer for quality.

Every change is followed by an evaluation run. A performance gain that breaches a quality threshold is rejected.

Practitioner approach

Measure before changing anything. Use telemetry to break a typical request into its steps (retrieval, each model call, each tool call, post-processing) with time and tokens for each, at the 50th, 90th and 99th percentiles. The slowest step is often not the one people assume; a sequential chain of three tool calls or an oversized retrieval payload can cost more than the main model call.

Set a latency budget per step that adds up to the target, and a token budget per call. Budgets turn vague complaints into engineering decisions and make regressions visible in the pipeline.

Remove before you optimise. Look for model calls doing work code could do (formatting, simple routing, extracting a field from structured data), duplicate retrieval, and agent loops that run more turns than needed. Each removed call saves seconds and cost, and removes a failure point.

Run independent work in parallel. Retrieval from several sources, or tool calls that do not depend on each other, can run concurrently. Check downstream rate limits before parallelising widely.

Stream responses where users read them. Time to first token matters more to perceived speed than total time. Where output must be validated before display, stream progress indicators or partial, safe content instead.

Trim tokens with care. Shorten instructions, cap conversation history with summarisation, and retrieve fewer, better chunks. Smaller prompts are faster and cheaper, but each trim can affect quality, so evaluate after each change.

Cache deliberately. Cache embeddings, retrieval results for stable content and responses to identical requests where appropriate, keyed so one user's permission-trimmed results are never served to another. Set expiry aligned with how often the content changes.

Right-size models per step. Routing, classification and extraction often run well on smaller, faster models; reserve larger models for steps that need them. Treat any model change as a release that re-runs the full evaluation.

Plan capacity. Translate expected peak load into tokens per minute and requests per minute per deployment, compare with quota, and decide on additional capacity, deployments or load spreading. Run load tests that reproduce realistic mixes of requests at expected peak and beyond, and watch for throttling and latency growth. Capacity commitments and their licensing implications go to the platform team and the Microsoft licensing guide or a licensing partner.

Outputs

• A performance baseline per step and percentile, and latency and token budgets.
• A tuning log: each change, its effect on latency, tokens, cost and evaluation scores, and the decision.
• A capacity plan: expected peak, required throughput per deployment, quota position and actions.
• Load test results against targets, as release evidence for tier 3 and above.

Microsoft platform considerations

Microsoft Foundry offers several deployment types with different throughput, latency and cost characteristics, and quotas and capacity are allocated per model, deployment and region. These options, their limits and their pricing change often; check ms-foundry-models and other records in references/sources.md and record what was confirmed before designing capacity. Never quote quotas or prices from memory.

Adoption considerations

Perceived speed shapes adoption more than measured speed. Streaming, progress indicators and honest messages when a request will take longer keep users engaged. Share improvements with the business owner in terms users notice.

Knowledge considerations

Retrieval payload size drives latency and tokens. Well-structured, concise knowledge with good metadata lets retrieval return fewer, better chunks; raise content issues with knowledge owners through knowledge-management:ai-ready-content.

Security and privacy considerations

Caches can leak data across users if keys ignore identity or permissions. Key caches on user or permission scope for user-scoped content, set expiry and protect cache stores like the data they hold.

Responsible AI considerations

Swapping in a smaller model or trimming context can reduce quality unevenly, for example for less common languages or complex cases. Check evaluation results by segment after performance changes, not only overall averages.

Evaluation requirements

Re-run the evaluation harness after every performance change, and reject changes that breach thresholds. Include load test results and latency at the agreed percentile in release evidence for tier 3 and above.

Value measures

Track latency at the agreed percentile, throttled requests, tokens and cost per request, and abandonment rates where measurable, at solution level. The AI engineering lead owns these, reporting to the business owner.

Scale notes

Starter

Most starter solutions need measurement, streaming and token trimming rather than capacity work. Instrument per-step latency and fix the slowest step first.

Scale

Set budgets per step, run load tests before wide rollouts, right-size models per step and plan quota with the platform team ahead of growth.

Enterprise

Manage capacity across solutions and regions, share caching and routing components, and include performance budgets in engineering standards with load testing in the pipeline for critical solutions.

Hand-offs

• To ai-foundry:foundry-cost-management with token and cost changes.
• To ai-foundry:foundry-deployment-architecture where deployment type or region changes are needed.
• To ai-engineering:ai-resilience for throttling and overload behaviour.
• To ai-engineering:evaluation-harness to confirm quality after each change.

Reviewer gates

From risk tier 3, design-authority checks performance evidence against agreed targets before production. A model change made for speed without a passing evaluation run is a blocking finding.

Failure modes

• A smaller, faster model is swapped in, latency improves and answer quality falls without anyone measuring it.
• A cache keyed only on the question serves one user's permission-trimmed answer to another.
• Load testing uses one repeated simple request, and real traffic throttles on day one of rollout.
• Tuning targets averages, while the slowest tenth of requests remain unusable.

Sources

references/sources.md in this skill lists the source records behind this skill, what each is authoritative for, how volatile it is and what you may do with it. Product availability, licensing, limits and preview status change often, so check the current source before stating them, and cite what you used.

Never

• Change a model or trim context for speed without re-running evaluation.
• Cache user-scoped results without keying on the user's identity or permissions.
• State quotas, throughput limits or prices from memory.
• Tune without per-step measurement and agreed targets.