Catalogue › AI Engineering

CopilotForge - AI Engineering

Use it in Copilot Cowork: download the AI Engineering plugin for Copilot Cowork and upload it under Customize > Plugins > Upload plugin. How to install.

Provide pro-code engineering patterns shared across custom AI solutions.

Layer: delivery
Lead role: AI Engineering Lead
Plugin name: copilotforge-ai-engineering

Skill Status Lifecycle Description

ai-api-and-tooling draft Deliver, Assure, Improve Designs the APIs and shared tooling that custom AI solutions call or expose - contract-first interfaces, narrow tool surfaces a model can use correctly, input validation, user-scoped authorisation, idempotent writes, rate limits, versioning and errors a model can act on. Use when building a service an agent will call as a tool, when exposing an AI capability as an API, or when shared tool libraries are needed across solutions.

ai-ci-cd draft Deliver, Assure, Launch, Improve Designs CI/CD pipelines for custom AI solutions that build, test, evaluate and deploy code, prompts, model configuration, tools and infrastructure together, with evaluation and safety gates, environment promotion, approvals, model version upgrades and rollback. Use when a pro-code AI solution needs a release pipeline, when prompts or models change outside the pipeline, or when a model version is being retired.

ai-engineering-standards draft Mobilise, Deliver, Assure, Scale, Improve Sets and maintains engineering standards for custom AI work - minimum practices by risk tier for repositories, identity, tools, testing, evaluation, pipelines, telemetry, performance, resilience and ownership - with a checklist, exception route and review cycle. Use when pro-code AI work starts, when teams build AI solutions inconsistently, or when standards need refreshing after platform change.

ai-performance-engineering draft Deliver, Assure, Scale, Improve Engineers the latency and throughput of custom AI solutions against agreed targets - latency budgets per step, token budgets, streaming, caching, parallel calls, right-sized models, quota and capacity planning, and load testing - while protecting quality and cost. Use when an AI solution is too slow, when usage is about to grow, when throttling appears under load, or before a tier 3 solution goes to production.

ai-resilience draft Deliver, Assure, Improve Engineers resilience into custom AI solutions - timeouts, bounded retries, circuit breakers, evaluated fallback deployments or models, degraded modes, content filter and malformed output handling, idempotent tools - so they fail safely and honestly. Use when building a pro-code AI solution, after an outage or throttling incident, or when nothing defines what happens if the model is unavailable.

ai-solution-repository draft Mobilise, Deliver, Improve Sets the repository structure for custom AI solutions so that code, system prompts, tool definitions, evaluation datasets, infrastructure as code and per-environment configuration are versioned together, owned and reviewable, with no secrets in source. Use when starting a pro-code AI solution, when prompts live outside source control, or when a team needs a standard repository template for Foundry or Azure AI work.

ai-solution-testing draft Deliver, Assure, Improve Sets the testing strategy for pro-code AI solutions - unit tests around non-model code, contract tests for tools, recorded or stubbed model responses, integration, adversarial and failure-path tests - and how each layer fits alongside evaluation. Use when planning tests for a Foundry or Azure AI solution, when tests are flaky because they call live models, or when testing stops at a few manual prompts.

ai-telemetry draft Deliver, Assure, Launch, Improve Designs the telemetry a custom AI solution emits - traces across orchestration, model, retrieval and tool calls, versions, tokens, latency, safety and quality signals and correlation ids - and decides what prompt and response content is captured, redacted and kept. Use when building a pro-code AI solution, when failures cannot be diagnosed, or when logs hold more personal data than they should.

evaluation-harness draft Deliver, Assure, Improve Builds the automated evaluation harness for a custom AI solution - dataset loading, repeatable runs against pinned models and prompts, code-based and model-graded evaluators, scoring against agreed thresholds, results history and reports - so quality is checked on every change. Use when an AI solution needs repeatable evaluation in its pipeline, when manual spot checks no longer scale, or when evaluation results cannot be reproduced.

secrets-and-identity draft Deliver, Assure, Improve Engineers the identities and secrets of a custom AI solution - managed identities and keyless access to model endpoints, on-behalf-of user access for data and tools, agent and workload identities, least-privilege role assignments, a managed secret store, rotation and secret scanning. Use when building a Foundry or Azure AI solution, when keys appear in code or configuration, or when an agent's service identity has broad access.

Skill	Status	Lifecycle	Description
`ai-api-and-tooling`	draft	Deliver, Assure, Improve	Designs the APIs and shared tooling that custom AI solutions call or expose - contract-first interfaces, narrow tool surfaces a model can use correctly, input validation, user-scoped authorisation, idempotent writes, rate limits, versioning and errors a model can act on. Use when building a service an agent will call as a tool, when exposing an AI capability as an API, or when shared tool libraries are needed across solutions.
`ai-ci-cd`	draft	Deliver, Assure, Launch, Improve	Designs CI/CD pipelines for custom AI solutions that build, test, evaluate and deploy code, prompts, model configuration, tools and infrastructure together, with evaluation and safety gates, environment promotion, approvals, model version upgrades and rollback. Use when a pro-code AI solution needs a release pipeline, when prompts or models change outside the pipeline, or when a model version is being retired.
`ai-engineering-standards`	draft	Mobilise, Deliver, Assure, Scale, Improve	Sets and maintains engineering standards for custom AI work - minimum practices by risk tier for repositories, identity, tools, testing, evaluation, pipelines, telemetry, performance, resilience and ownership - with a checklist, exception route and review cycle. Use when pro-code AI work starts, when teams build AI solutions inconsistently, or when standards need refreshing after platform change.
`ai-performance-engineering`	draft	Deliver, Assure, Scale, Improve	Engineers the latency and throughput of custom AI solutions against agreed targets - latency budgets per step, token budgets, streaming, caching, parallel calls, right-sized models, quota and capacity planning, and load testing - while protecting quality and cost. Use when an AI solution is too slow, when usage is about to grow, when throttling appears under load, or before a tier 3 solution goes to production.
`ai-resilience`	draft	Deliver, Assure, Improve	Engineers resilience into custom AI solutions - timeouts, bounded retries, circuit breakers, evaluated fallback deployments or models, degraded modes, content filter and malformed output handling, idempotent tools - so they fail safely and honestly. Use when building a pro-code AI solution, after an outage or throttling incident, or when nothing defines what happens if the model is unavailable.
`ai-solution-repository`	draft	Mobilise, Deliver, Improve	Sets the repository structure for custom AI solutions so that code, system prompts, tool definitions, evaluation datasets, infrastructure as code and per-environment configuration are versioned together, owned and reviewable, with no secrets in source. Use when starting a pro-code AI solution, when prompts live outside source control, or when a team needs a standard repository template for Foundry or Azure AI work.
`ai-solution-testing`	draft	Deliver, Assure, Improve	Sets the testing strategy for pro-code AI solutions - unit tests around non-model code, contract tests for tools, recorded or stubbed model responses, integration, adversarial and failure-path tests - and how each layer fits alongside evaluation. Use when planning tests for a Foundry or Azure AI solution, when tests are flaky because they call live models, or when testing stops at a few manual prompts.
`ai-telemetry`	draft	Deliver, Assure, Launch, Improve	Designs the telemetry a custom AI solution emits - traces across orchestration, model, retrieval and tool calls, versions, tokens, latency, safety and quality signals and correlation ids - and decides what prompt and response content is captured, redacted and kept. Use when building a pro-code AI solution, when failures cannot be diagnosed, or when logs hold more personal data than they should.
`evaluation-harness`	draft	Deliver, Assure, Improve	Builds the automated evaluation harness for a custom AI solution - dataset loading, repeatable runs against pinned models and prompts, code-based and model-graded evaluators, scoring against agreed thresholds, results history and reports - so quality is checked on every change. Use when an AI solution needs repeatable evaluation in its pipeline, when manual spot checks no longer scale, or when evaluation results cannot be reproduced.
`secrets-and-identity`	draft	Deliver, Assure, Improve	Engineers the identities and secrets of a custom AI solution - managed identities and keyless access to model endpoints, on-behalf-of user access for data and tools, agent and workload identities, least-privilege role assignments, a managed secret store, rotation and secret scanning. Use when building a Foundry or Azure AI solution, when keys appear in code or configuration, or when an agent's service identity has broad access.

Credit

Created by Simon Foote as part of CopilotForge. Content is licensed under CC BY 4.0; keep the credit in each skill's license and authors fields when you share or adapt it. See NOTICE.md in the repository.