Governing Agentic AI
Executive Summary
In September 2025, hackers deployed the first-documented autonomous AI cyberattack, targeting 30 organisations using Anthropic’s Claude platform . The attack was almost 90% autonomous.
The capabilities that enabled this attack—autonomous decision-making, adapting to obstacles, coordinating across systems—didn’t exist a year ago. Boards are increasingly approving agentic systems for customer service, supply chain, and financial operations.
The advice from AI practitioners for governing agentic AI : deploy selectively, define boundaries, and build foundations first.
The Governance Problem
Your company will likely be deploying agentic AI in the coming year, if you have not already. This shift from generative AI (responds to prompts) to agentic AI (acts autonomously to achieve goals) has outpaced board governance and risk frameworks.
Leading platforms from Amazon, Google, Microsoft, Nvidia, and Salesforce now embed agentic capabilities—systems that run in loops and make decisions with minimal human input. They deliver powerful efficiency gains but also come with unpredictable risks.
“Every time an AI makes a decision about your life, it’s doing maths it can’t explain, following logic it doesn’t understand, creating outcomes nobody can fully trace,” explains Nita Farahany, Professor of Law and Philosophy at Duke University.
Agents amplify risk
Agentic AI represents a fundamental governance reset, not just an incremental risk. AI agents work in multi agent systems that both amplify single-agent failures and generate entirely new failures that come from the interactions between agents.
“A collection of safe agents does not make a safe collection of agents” is the core risk identified in a report on multi-agents by the Gradient Institute
Read the full report at Gradient Institute
Three Practitioner Perspectives
I consulted three expert AI practitioners on how boards can govern autonomous systems
Pascal Biese - AI Engineer at PwC Belgium, founder of LLM Watch (36k subscribers)
Devansh - Engineer, founder of AI Made Simple (238k subscribers)
Lawrence Puang - Data Scientist and CEO of Xephyr AI
Their recommendations focus on selective deployment, operational boundaries and organisational readiness.
Deploy Agentic AI Sparingly
“The most useful question isn’t ‘can we use agentic AI for this’ but ‘should we?’ If a process is predictable, you won’t need autonomous agents. A well-defined workflow will be a better choice.”
Pascal Biese states.
Autonomous AI should be reserved for genuinely dynamic situations, such as real-time pricing, where instant, human-like decision-making adds measurable value.
“If a process is deterministic, you won’t need agents,” Biese explains. “If it’s partly deterministic, partly dynamic, then only use agentic AI for the dynamic bits.”
Limit the Blast Radius
Devansh Devansh
“Scope agents rigorously. Make sure you define how the agents will work. Limit blast radius and ensure monitoring.”
Devansh recommends
Boards need to quantify the maximum potential impact from AI decisions and establish clear boundaries.
Define not just what agentic AI can do, but what it cannot do under any circumstances Pascal Biese suggests
Biese poses a series of questions to test when deploying agentic ai.
1. What’s the maximum potential loss from a single agent decision? Quantify scenarios in financial, reputational, and operational terms. Consider compound effects from chains of automated decisions.
2. How do we bound the operational scope? What are the explicit limits on agent actions, resource access, and decision authority? Can the agent modify its own objectives or spawn sub-agents?
3. How do agent failures propagate through our systems? Map interdependencies between AI agents and critical business processes.
4. What level of autonomy are we granting, and where are the human checkpoints? Map out exactly which decisions agents can make independently versus those requiring human approval. Define clear escalation triggers.
5. What are our kill switches and circuit breakers? Ensure immediate shutdown capabilities that don’t create additional risks or data loss.
6. How do we detect when agents deviate from intended behaviour? Implement continuous monitoring for objective drift, unexpected emergent behaviours, and performance degradation.
Build Foundations First
Laurence Puang
“Most organisations fail to realise AI implementation because their data foundations are not ready, this is the most common failure we see” Lawrence Puang explains.
“Directors should have visibility on all AI use cases in the enterprise with an AI scorecard for each—showing outcomes, purpose, risks, metrics, guardrails, data usage, models, and controls.”
Boardroom Checklist
Based on my consultations with Pascal Biese, Devansh, and Lawrence Puang, here are specific questions
Strategic Restraint
✓ Do we even need agents for this task?
✓ What level of autonomy are we granting, and where are the human checkpoints?
✓ Could a simpler, deterministic system achieve the same outcome?
✓ Which parts of this process are predictable vs. dynamic?
Risk Containment
✓ What’s the maximum potential loss from a single agent decision?
✓ How do we bound the operational scope? What are the explicit limits?
✓ Can the agent modify its own objectives or spawn sub-agents?
✓ How do agent failures propagate through our systems?
✓ What are our kill switches and circuit breakers?
✓ How do we detect when agents deviate from intended behaviour?
Organisational Readiness
✓ Do we have an AI scorecard showing all use cases, risks, and controls?
✓ Are our data foundations and governance structures adequate?
✓ Do we have AI-literate leadership in the C-suite and board?
✓ Can we monitor agentic AI behaviour in real-time?
✓ Have we simplified multi-agent interactions to prevent cascading issues?
Next steps
The three practitioners I consulted were clear that restraint in using agentic AI beats enthusiasm. Deploy sparingly, bound tightly, and build monitoring before you scale. Governing an unpredictable technology is about asking the right questions. Boards need to keep asking questions about the technology, where it is used and how risks are being managed. Setting up an AI scorecard showing what’s already deployed and assessing risk mitigation is the a good place to start.