Governance-by-Design for Trusted AI Operations

Enterprise AI adoption has outpaced governance. Models are deployed into customer-facing workflows, agentic systems are granted tool access, and regulators have moved from guidance to enforceable obligations — yet most organizations still treat governance as a final review gate rather than an architectural input. This article unpacks the latest research on the governance gap, defines what "governance-by-design" actually means in practice, and lays out a framework for embedding security, compliance, observability, and oversight into AI systems from the first sprint.

The Governance Gap, in Numbers

Adoption is widespread; disciplined governance is not. McKinsey's State of AI in 2025 survey of 1,993 respondents across 105 nations finds that 88 percent of organizations now use AI in at least one business function, up from 78 percent a year earlier. Yet the same research shows that risk mitigation remains uneven: organizations are now actively managing an average of four AI-related risks, up from two in 2022, and 51 percent report at least one negative consequence from AI use — with inaccuracy the most commonly cited.

The accountability gap is sharper. McKinsey finds that CEO oversight of AI governance is one of the elements most correlated with higher self-reported bottom-line impact from generative AI, yet only 28 percent of respondents say their CEO owns AI governance, and just 17 percent report board-level oversight. In many cases governance is jointly owned but unclearly divided — respondents report an average of two leaders in charge, without always specifying decision rights.

McKinsey's 2026 AI Trust Maturity Survey, covering approximately 500 organizations with direct responsibility for AI governance or risk, reinforces the pattern. Organizations with explicit accountability for responsible AI — through dedicated governance roles, internal audit, or ethics teams — score materially higher on trust maturity (average 2.6 vs. 1.8 for those without clear ownership). Knowledge and training gaps remain the leading barrier to responsible AI implementation. Only about 30 percent of organizations score at maturity level three or higher on agentic AI governance and controls — despite 65 percent naming security and risk concerns as the biggest barrier to scaling agentic AI.

As AI systems become more autonomous and embedded in critical workflows, gaps in governance and risk management will become increasingly costly. — McKinsey, State of AI Trust in 2026

The pattern across these studies is consistent: AI is being deployed faster than it is being governed. Closing that gap is not primarily a policy-writing problem. It is an architecture and operating-model problem.

Why Bolt-On Governance Fails

Gartner's July 2024 projection that at least 30 percent of generative AI projects will be abandoned after proof of concept by end of 2025 explicitly cites "inadequate risk controls" alongside poor data quality and unclear business value. The controls fail not because teams ignore risk, but because governance is applied after the architecture is set.

Five recurring patterns explain why retrofitted governance rarely holds:

1. The policy perimeter does not match the system boundary

Legal and compliance teams review a use-case brief while engineering has already chosen a vendor API, connected production data sources, and granted tool access. When the policy finally arrives, it describes a system that does not exist — or it describes constraints that require a rewrite.

2. Security is treated as a model feature, not a system property

Teams assume the foundation model's safety training is sufficient. It is not. The OWASP Top 10 for LLM Applications 2025 ranks prompt injection (LLM01) as the #1 risk for the second consecutive release, noting that there is no universal fix — only defense-in-depth. Prompt injection exploits the architecture itself: instructions and data share the same channel. OWASP also highlights sensitive information disclosure (LLM02), supply chain vulnerabilities (LLM03), excessive agency (LLM06) for agentic systems, and vector and embedding weaknesses (LLM08) for RAG deployments — each requiring system-level controls, not model-level hope.

3. Observability stops at the API call

Logging a request ID and token count is not governance telemetry. Audit-ready systems need step-level traces: what context was retrieved, which tools were invoked, what policy checks ran, and what a human reviewer saw. Without that granularity, incident response and regulatory inquiry both degrade into forensic archaeology.

4. Human oversight is designed as a fallback, not a control

NIST's Generative AI Profile (AI 600-1) identifies human-AI configuration — automation bias, overreliance, and inappropriate anthropomorphizing — as a distinct GenAI risk class. Human-in-the-loop that activates only after a bad output reaches a customer is not oversight; it is damage control.

5. Regulatory artifacts are produced once, then orphaned

A risk assessment written at launch and never updated is worse than no assessment — it creates a false sense of compliance. The EU AI Act, NIST AI RMF, and ISO/IEC 42001 all assume lifecycle governance: controls that evolve as the system, its data, and its deployment context change.

All five patterns share a structural cause: governance is treated as a *document* rather than as a *plane* that spans the system. Governance-by-design is what you get when the plane is built in, not pasted on.

What "Governance-by-Design" Actually Means

Governance-by-design embeds risk controls, auditability, and oversight into the architecture and delivery workflow of an AI system — so that compliance artifacts are a by-product of normal operation, not a separate remediation project. The definition has four parts:

• Policy-as-architecture. Allowed data sources, retention periods, tool permissions, and output constraints are enforced in code — retrieval filters, API scopes, output validators — not only in policy PDFs.
• Continuous measurement. The system is evaluated against safety, accuracy, and fairness criteria on a defined cadence, with offline harnesses in CI and online sampling in production.
• Traceable by default. Every interaction produces an audit record sufficient to answer "what happened, with what data, under which policy version, and who approved it?"
• Human authority preserved. High-impact actions require explicit human approval; deactivation criteria and escalation paths are documented and tested.

This aligns with NIST's framing. The AI RMF 1.0 Core organizes practice around four functions — GOVERN, MAP, MEASURE, MANAGE — across 19 categories and 72 subcategories. GOVERN is cross-cutting; MAP, MEASURE, and MANAGE apply per system and per lifecycle stage. NIST emphasizes these are not a checklist but an iterative cycle. A Gartner survey of 360 organizations found that those deploying AI governance platforms are 3.4× more likely to achieve high effectiveness in AI governance than those without — suggesting that tooling which enforces controls at runtime, not just documents them, materially changes outcomes.

Agentic AI Governance: When Systems Act, Not Just Answer

The shift to agentic AI changes the governance equation fundamentally. In McKinsey's State of AI Trust in 2026, Partner Rich Isenberg frames it precisely: "Agency isn't a feature — it's a transfer of decision rights." The question moves from "Is the model accurate?" to "Who is accountable when the system acts?" Production governance for agents requires four mechanisms that bolt-on policies cannot supply:

• Agent inventory and identity binding. Every agent must be cataloged with defined scope, access rights, and a single accountable human owner — not a team, not a department, a person. "You can't govern what you can't see," as McKinsey puts it.
• Autonomy tiers. Risk-based classification that dictates human oversight levels: read-only agents, draft-and-approve agents, and fully autonomous agents with action budgets and irreversibility gates.
• Embedded control agents. Guardrails and compliance checks wired into the execution pipeline — not periodic audits after the fact. Policy enforcement must generate proof at the moment an action occurs.
• End-to-end traceability. The ability to reconstruct any agent decision: which tools were invoked, under which policy version, with what retrieved context, and who approved escalation.

McKinsey's 2026 AI Trust Maturity Model adds agentic AI governance and controls as a fifth dimension alongside strategy, risk management, data and technology, and governance — reflecting that autonomous systems are now a distinct governance surface, not a subset of chatbot policies.

EnterpriseDB's Governing Agentic AI at Enterprise Speed framework makes the data-layer case explicit: policies defined at the business layer are invisible to agents in motion, so enforcement must live where data is accessed. Its three-phase roadmap — Foundation (read-only, well-understood data), Expansion (reversible write actions with intent validation), and Scale (customer-facing and systems-of-record with anomaly detection and full session reconstruction) — aligns with the autonomy tiers above and provides a practical sequencing model for teams building governance architecture before agent scope expands.

Reference Frameworks: The De Facto Baseline

Three frameworks have converged into the baseline that enterprise buyers, auditors, and regulators now expect. Teams that ignore them accumulate rework that compounds as systems scale.

NIST AI Risk Management Framework

Released January 26, 2023, NIST AI RMF 1.0 is voluntary U.S. guidance that has effectively become common vocabulary for AI risk. NIST provides a companion AI RMF Playbook, crosswalks, and a Trustworthy and Responsible AI Resource Center launched March 30, 2023. In April 2026, NIST released a concept note for an AI RMF Profile on Trustworthy AI in Critical Infrastructure.

NIST Generative AI Profile (AI 600-1)

Released July 26, 2024, NIST AI 600-1 maps 12 GenAI-specific risks — including confabulation, data privacy, information integrity, harmful bias, human-AI configuration, information security, and intellectual property — into the four RMF functions with more than 200 suggested actions. Actions marked "foundational" are treated as minimum tasks for GenAI risk management.

ISO/IEC 42001:2023

Published December 18, 2023, ISO/IEC 42001 is the first international, certifiable standard for an Artificial Intelligence Management System (AIMS). It follows the same Annex SL structure as ISO/IEC 27001. Annex A defines 38 AI-specific controls across nine objectives (A.2 through A.10), covering policy, impact assessment, lifecycle, data, third-party risk, and responsible use. Organizations select applicable controls via a Statement of Applicability (SoA) — the document auditors and enterprise buyers most often request.

EU AI Act

Regulation (EU) 2024/1689 — the EU AI Act — entered into force August 1, 2024. Key deadlines already binding or imminent:

• 2 February 2025 — prohibited AI practices (Article 5) and AI literacy obligations (Article 4) became enforceable. Penalties for prohibited-practice violations reach €35 million or 7 percent of global annual turnover.
• 2 August 2025 — general-purpose AI model obligations (Articles 51–56) became binding, including technical documentation, training-data summaries, and downstream information sharing.
• 2 December 2026 — under the Digital Omnibus provisional agreement reached May 7, 2026, watermarking and synthetic-content disclosure obligations under Article 50(2) take effect, along with a new prohibition on AI "nudification" systems. Any generative feature in the EU market needs labelling, metadata embedding, and detection capability operational by this date.
• 2 December 2027 / 2 August 2028 — high-risk obligations for Annex III standalone systems and Annex I embedded systems, respectively, under the Omnibus timeline (postponed from the original 2026–2027 dates).

Practically, an AI system designed in 2026 should already produce the artifacts these frameworks require: documented risk assessment, data governance records, lineage and logs for post-market monitoring, human-oversight design for high-risk use cases, and a SoA or equivalent control mapping. Producing them retroactively is significantly more expensive.

Security by Design: The OWASP LLM Top 10

The OWASP Top 10 for LLM Applications 2025 translates application security discipline to GenAI. The full 2025 list — LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM03 Supply Chain, LLM04 Data and Model Poisoning, LLM05 Improper Output Handling, LLM06 Excessive Agency, LLM07 System Prompt Leakage, LLM08 Vector and Embedding Weaknesses, LLM09 Misinformation, LLM10 Unbounded Consumption — maps directly to governance-plane components. The 2025 release expands scope for agentic systems (LLM06) and RAG deployments (LLM08), and adds system prompt leakage (LLM07) as a distinct attack class.

For governance-by-design, three implications matter most:

• Prompt injection is architectural. OWASP notes there is no foolproof prevention — only mitigation through privilege separation, input/output filtering, and human approval for high-risk actions. The LLM should operate with minimal API permissions, not the user's full credentials.
• Supply chain is a governance problem. Third-party models, embeddings, plugins, and training data each introduce trust boundaries. ISO 42001 control A.10 and OWASP LLM05 both require documented third-party assessment and ongoing monitoring.
• Agentic systems expand the blast radius. Excessive agency — granting an LLM the ability to take consequential actions without appropriate constraints — is now a top-ten risk. Agent governance requires tool allowlists, action budgets, mandatory human gates for irreversible operations, and per-agent identity binding so actions are attributable.
• Red teaming is continuous, not annual. OWASP and NIST both recommend adversarial testing on every prompt, model, retrieval, or tool-permission change — integrated into CI alongside the evaluation harness, not relegated to an annual penetration test.

A Reference Architecture for the Governance Plane

Governance-by-design requires a horizontal governance plane that spans the context, model, orchestration, and observability layers. Six components are usually present in production systems:

Policy engine

Encodes allowed data classifications, tool permissions, output formats, and geographic constraints. Evaluated before retrieval, before model calls, and before tool execution. Changes are versioned and require approval.

Identity and entitlements

Users see only the documents and tools they are authorized to access. Retrieval indexes inherit source-system entitlements; the model never receives data the user could not fetch directly.

Safety and content filters

Input classifiers for injection and jailbreak patterns; output filters for PII, regulated content, and policy violations. Filters run deterministically where possible, with model-based judges for edge cases.

Observability and audit log

Step-level tracing with immutable audit records: prompt template version, model version, retrieved document IDs, tool calls, policy decisions, and final output. Retention aligned to regulatory requirements.

Evaluation harness

Offline datasets (golden, regression, adversarial, fairness) run in CI on every prompt, model, or retrieval change. Online evaluation samples production traffic and feeds regressions back into the offline set.

Human oversight workflow

Queue for high-risk outputs, low-confidence responses, and policy exceptions. Reviewer actions are logged; feedback updates the evaluation set. Deactivation criteria — when to pull a model or prompt — are documented per NIST AI 600-1 Manage guidance.

Observability, Audit, and Incident Response

Trusted AI operations require the same incident discipline as production software — adapted for probabilistic failure modes:

• Silent failures are the primary risk. A 2 percent regression in grounding accuracy may matter more than a 500-error spike. SLOs should include accuracy, hallucination rate, refusal rate on disallowed requests, and policy violation rate — not only latency and availability.
• Drift is monitored in both directions. Input drift, context drift (underlying documents change), and output drift (vendor silently updates the model) each need thresholds and named owners.
• Incidents are practiced. Runbooks for hallucination spikes, retrieval index corruption, prompt injection in user content, model vendor outage, and regulator inquiry are written, owned, and exercised.
• Audit queries are rehearsed. Before production, the team should demonstrate it can answer "what did the system tell customer X on date Y?" and "which policy version was active?" in minutes.

The Operating Model: Roles, Rituals, and Decision Rights

Architecture enables governance; the operating model sustains it. Four roles are essential — they need not be four people, but each function requires a named owner:

• Governance/risk lead — accountable for the policy perimeter, regulatory artifacts, SoA, and model/prompt review process.
• AI engineering lead — accountable for the evaluation harness, safety filters, and release gating. Owns accuracy and safety SLOs.
• Platform/security lead — accountable for entitlements, audit logging, incident response, and third-party model risk.
• Product/value owner — accountable for use-case approval, human-oversight design, and deactivation authority when business risk exceeds tolerance.

Three rituals operationalize the roles:

• A weekly evaluation review covering regressions, new adversarial examples, pending releases, and policy changes.
• A monthly risk and incident review covering drift, user feedback, third-party model updates, and corrective actions.
• A quarterly control review updating the risk assessment, SoA, and regulatory artifacts as systems and obligations evolve.

Four decision rights need explicit owners and recorded SLAs: go-live, model swap, policy change, and emergency deactivation/rollback.

A 90-Day Path to Governance-by-Design

For a pilot entering production, a focused 90-day program establishes a defensible governance baseline:

Days 1–30: Map and instrument

Complete a system-level risk assessment mapped to NIST AI RMF and OWASP LLM Top 10. Instrument step-level tracing and audit logging. Define the first version of the policy engine (data allowlists, tool permissions, output constraints). Exit criterion: every production request produces an audit record.

Days 31–60: Measure and gate

Stand up the evaluation harness with golden, adversarial, and fairness datasets. Gate releases on regression thresholds. Implement human-oversight queues for high-risk actions. Produce first-version regulatory artifacts: risk assessment, data sheet, human-oversight design. Exit criterion: no prompt or model change reaches production without passing the harness.

Days 61–90: Certify and operationalize

Complete the Statement of Applicability (or equivalent control mapping). Run an end-to-end incident exercise and audit query drill. Establish weekly evaluation and monthly risk reviews as recurring rituals. Exit criterion: a named owner can produce audit-ready documentation for any production interaction within one business day.

Trust as a Speed Enabler

McKinsey's State of AI Trust in 2026 research increasingly frames responsible AI as a business enabler rather than a compliance tax — particularly as agentic systems gain tool access and autonomy. Organizations with explicit accountability, embedded controls, and continuous measurement deploy faster over time because each release reuses proven governance patterns.

The contrast is stark: organizations that score high on policy documents but lack runtime enforcement discover this at the worst moment — when a regulator, customer, or board asks for proof that governance was applied at the time an agent acted, not just that a policy existed in a wiki. Governance-by-design produces that proof as a by-product of normal operation.

The governance moat in enterprise AI is not a better policy document. It is the system around the model — policy enforced in architecture, risk measured continuously, and oversight wired into the workflow — that lets teams scale AI without scaling regret.

References

• McKinsey & Company. *The State of AI in 2025: Agents, Innovation, and Transformation.* November 2025. mckinsey.com
• McKinsey & Company. *State of AI Trust in 2026: Shifting to the Agentic Era.* January 2026. mckinsey.com
• EnterpriseDB. *Governing Agentic AI at Enterprise Speed.* 2026. enterprisedb.com
• Gartner. *Gartner Predicts 30% of Generative AI Projects Will Be Abandoned After Proof of Concept By End of 2025.* Press release, July 29, 2024. gartner.com
• NIST. *Artificial Intelligence Risk Management Framework (AI RMF 1.0).* January 26, 2023. nist.gov
• NIST. *AI 600-1: Generative Artificial Intelligence Profile.* July 26, 2024. nvlpubs.nist.gov
• ISO/IEC. *ISO/IEC 42001:2023 — Artificial Intelligence Management System.* December 18, 2023. iso.org
• European Union. *Regulation (EU) 2024/1689 (Artificial Intelligence Act).* In force August 1, 2024. artificialintelligenceact.eu
• Council of the EU. *Artificial Intelligence: Council and Parliament agree to simplify and streamline rules.* Press release, May 7, 2026. consilium.europa.eu
• OWASP. *Top 10 for LLM Applications 2025.* owasp.org