The AI CoE is the central, cross-functional team that owns the AI program. Every source article — Microsoft CAF, lakeFS, IBM, Automation Anywhere, Elementum, Databricks — converges on the same idea: a centralized hub of expertise, standards, governance, and enablement, operating across business units.
The CoE is not an ivory tower. Its job is to make the right thing the easy thing for the rest of the company.
6.1 Five CoE operating models
Pick the one that matches the company's size and maturity. We start in one and evolve.
| Model | When it fits | Risk |
|---|---|---|
| Centralized | Early-stage AI adoption. Few use cases. Tight control needed. CoE does most of the building. | Becomes a bottleneck as demand from departments grows. |
| Federated | AI capability spread to business units; light central function for standards. | Hard to keep consistency without strong coordination. |
| Hybrid | Central CoE sets standards and tooling; business units build use cases on the shared platform. | The most commonly recommended model. Requires real platform investment. |
| Platform-led | CoE's primary product is an internal AI platform; teams self-serve. | Only works if the platform is genuinely easier than going around it. |
| Domain-focused | CoE specialized to one function (Finance, Operations, etc.). | Risks silos and duplicated work across other domains. |
Default recommendation: start Centralized. Evolve to Hybrid as agent count grows past ~10 in production. Move toward Platform-led / Advisory once the platform is mature.
6.2 What the CoE does
Concrete responsibilities, drawn from Microsoft CAF + Automation Anywhere + IBM:
| Area | Responsibility |
|---|---|
| Strategy | Define AI strategy aligned to business goals. Maintain the AI portfolio. Identify use cases. |
| Skills | Assess AI literacy across the org. Run training, hackathons, hands-on workshops. Build a champion network. |
| Intake | Operate a single intake process. Triage and prioritize requests via a Value-vs-Risk matrix. Maintain the backlog. |
| Standards | Define approved models, frameworks, tools, prompt patterns, evaluation criteria, security baselines. Audit compliance. |
| Pilot projects | Run strategic pilots. Use them to validate the framework itself, not just the agent. |
| Reusable assets | Curate prompt templates, agent blueprints, RAG patterns, evaluation harnesses, compliance checklists. Version and deprecate them. |
| Measurement | Define KPIs. Track adoption, ROI, incidents, compliance. Report to leadership. |
| Operations (early) | Run the registry. Manage shared infrastructure. Provide on-call for shared services. |
6.3 CoE evolution: centralized → advisory
The CoE's role shifts as adoption matures. Failing to evolve is one of the most common structural failures (alongside not staffing a CoE at all).
| Stage | CoE behavior |
|---|---|
| Stage 0 — Foundation | Stand up CoE. Define standards. Run first 2–3 pilots end-to-end as the central team. |
| Stage 1 — Centralized | CoE builds everything. Strong consistency, growing backlog. |
| Stage 2 — Hybrid | CoE owns platform, registry, standards. Department champions build agents using shared blueprints. |
| Stage 3 — Advisory | CoE sets policy, audits compliance, runs incident response, manages portfolio. Platform team enforces guardrails. Workload teams own delivery. |
Triggers to move from one stage to the next: approval delays, growing friction between CoE and product teams, more requests than the CoE can serve.