AI governance framework: How enterprises govern autonomous agents
Tempo Team
Key Takeaways
A complete AI governance framework governs two halves, the model's safety and the work your agents do in the portfolio.
The standards (NIST, the EU AI Act, ISO 42001) govern model risk and stop short of whether an agent's work still serves the outcome it was funded for.
Govern agents work the way you govern human work, by what each initiative consumes and what it delivers.
Search “AI governance framework” and you get the same checklist: Is the model fair? Is it secure and lawful? That checklist matters, but it’s only half the story.
An AI governance framework is the system of principles, roles, controls, and oversight that decides how an organization builds and runs AI. It also specifies who is accountable when the AI acts.
When AI only produced predictions, model governance often sufficed: A person read each output and decided what to do with it. Autonomous agents are a lot more complicated – they remove that “human gate.” They plan and act continuously, like small teams.
That autonomy is already running in your portfolio. Tempo’s 2026 State of SPM report found 30.3% of top-performing teams use AI extensively, while none of the lowest performers do. The open question is who governs what they do. A complete AI governance framework must therefore cover two linked domains:
The model: Safety, fairness, security, and legal compliance.
The work: What the agent is authorized to do, how its tasks are funded and staffed, how its outputs map to approved outcomes, and who owns the results.
This guide explains the governance pillars, agent-specific controls, and the portfolio-level lens that ties agent work and spend to measurable results.
What an AI governance framework is
An AI governance framework is the system of controls and accountable owners that governs how your organization builds and runs AI. It makes two things explicit that policy memos often leave vague: Who is accountable for a given AI decision, and what the system must satisfy before and after it ships.
For most of the last decade, governance focused on models (training data, bias, accuracy, security) under the assumption a human decided what to do with the model's output.
Agentic AI (software that plans and carries out multi-step work on its own) removed that assumption. Now the framework has to account for a worker that acts without waiting for you, which is why governing the model alone leaves the most expensive questions unanswered: What is all this agent work costing, and is it working?
Why governing autonomous agents is different
A model returns an answer and waits. An agent decides and acts on its own, then does it again, often hundreds of times a day, without writing a status update.
That shift from output to action is what separates an AI agent governance framework from the model governance most enterprises already run.
This is because an agent acts on its own. It approves and commits work without waiting. So the oversight function moves from whether the output is accurate to whether the agent is allowed to act at all.
An agent consumes resources around the clock. It uses compute budget and takes on work a human used to do. Put simply, it takes up capacity in your portfolio whether or not you make provisions for it.
An agent runs continuously, so problems a quarterly review would have caught now build up between reviews, because the agent doesn't stop to wait for the next governance cycle.
The security field has moved fast on the first point. Frameworks now treat each agent as its own identity with least-privilege access, runtime guardrails, the same controls you would put on any non-human account with the authority to act.
What almost no framework handles is the second and third points together: An agent that continuously consumes capacity and budget, against initiatives someone funded for a specific result. That is a portfolio problem, and it needs portfolio governance.
The core pillars of an AI governance framework
AI governance frameworks rest on five recognized pillars. They are accountability, transparency, fairness, privacy and security, and human oversight.
Some version of these principles shows up in every credible framework, from the OECD's AI Principles to the EU AI Act. They are the values the rest of the structure exists to enforce, but a framework built for autonomous agents needs a sixth, portfolio accountability, for the work the agents do.
This is how they compare:
Pillar | What it governs | What it looks like in practice |
Accountability | Who owns each AI decision | Named owners: A model owner, a risk officer, an AI steering committee |
Transparency | Whether you can explain a decision | Decision logs, model documentation, traceable reasoning |
Fairness | Whether the system treats people equitably | Bias testing before and after deployment |
Privacy and security | How data and access are protected | Data controls, least-privilege access, adversarial testing |
Human oversight | Where a person stays in control | Approval thresholds set for high-stakes actions |
Portfolio accountability | Whether agent work maps to funded outcomes | Capacity, cost, and outcome tracked per initiative |
With portfolio accountability as part of the core pillars of AI governance, you’re creating a system that governs the work itself. It tracks the capacity and budget an agent consumes, and whether the initiative it is working on still delivers the outcome it was funded for.
You govern human work this way already. The section below extends it to agents.
The standards behind every framework: NIST, the EU AI Act, and ISO 42001
Enterprises rarely write a framework from scratch. They adopt one of three reference points and adapt it, and you should know each by name, because your auditors and your board are already familiar with these rules.
Standard | What it is | What it governs | Status |
NIST AI RMF | US voluntary risk framework, four functions | Model risk across the lifecycle | Voluntary |
EU AI Act | EU law, risk-tiered by harm | High-risk AI, including employment decisions | Binding, phasing in |
ISO/IEC 42001 | Certifiable management-system standard | The governance system itself | Certifiable |
1. The NIST AI Risk Management Framework is the US benchmark, built on four functions: Govern, Map, Measure, and Manage. NIST released version 1.0 in 2023 and added a Generative AI Profile in July 2024 for the risks generative systems bring. It is voluntary, which is why it spreads. Teams use it as the skeleton other requirements hang on.
2. The EU AI Act is the first binding AI law, and it is the standard most relevant to agents in the portfolio. It sorts systems into risk tiers, and it classifies AI used for "employment, worker management and access to self-employment" as high-risk, the same tier as credit scoring and critical infrastructure. If your agents make or shape decisions about who does what work, this reaches you.
3. ISO/IEC 42001 is the international, certifiable standard for an AI management system, the closest thing to an audit-ready stamp that you govern AI on purpose.
Notice what all three govern: The model and its risk. None asks whether the work an agent is doing still maps to the outcome it was funded for. They were not built to, and that is where your governance has to go further.
The controls for governing autonomous agents
Governing the model is table stakes. Governing the agent means controlling what it is allowed to do while it does it. Five controls do most of the work, and they map onto the security guidance that already exists for non-human accounts.
1. Scope and authority
Define what the agent is allowed to achieve and what it must never do, then separate the actions it can take on its own from the ones that need a human to approve them.
2. Identity and least privilege
Give each agent its own system identity instead of letting it run on a person's credentials, and grant only the access its task needs. An agent with "root" access can reach every system and change anything, so one compromised agent can lead to a full breach. The least-privilege access model limits what any one agent can do, so a mistake or compromise stays contained.
3. Runtime guardrails
Enforce limits while the agent runs, not only at design time, so an unsafe action gets blocked in the moment instead of discovered in a log afterward.
4. Logging and traceability
Record every decision and tool call, so any action can be reconstructed. This is also what an ISO 42001 audit and your EU AI Act obligations will ask for.
5. A kill switch
Keep a way to suspend an agent the instant it drifts, the operational version of pulling a breaker. Underneath those controls sits the real design decision: How much autonomy do you grant, and where?
The clearest way to think about governing autonomous agents is a ladder
At the observe level, the agent surfaces information and you decide
Move it up to assist, and it recommends an action for you to approve
At the delegate level, it carries out pre-approved actions on its own, inside guardrails you set
You move an agent up the ladder as it earns trust on a specific task, and the consequential calls stay with the people accountable for them no matter how high it climbs.
The missing section: Governing AI's work across the portfolio
Every framework tells you to build an AI inventory, a registry of every model and agent running in your organization, so nothing operates in the shadows.
Do it. An inventory is necessary, and it is only the starting point, because a registry tells you an agent exists. It cannot tell you what that agent is costing against a specific initiative, or whether that initiative is still delivering the outcome it was funded for.
That second question is portfolio governance, and it is the part the standards leave out. You already govern human work this way. When a team takes on a project, you track the capacity it uses, the cost it runs up, and the outcome it produces.
Agent work needs the same treatment, for the same reason: An initiative does not care whether a human or an agent did the work, only whether the work moved the outcome and what it cost to get there.
Here is the difference in practice.
An AI inventory tells you | Portfolio governance tells you |
This agent exists and who owns it | What this agent costs against each initiative |
What model and tools it uses | Whether that initiative is still delivering |
Its risk tier and access scope | How its capacity competes with the rest of the portfolio |
The shape of this already exists for human work. Tempo Structure PPM gives you one portfolio view of every project and program in Jira, where all the work, human and agent, rolls up. The same Jira-native suite already tracks what that work costs and the capacity it consumes, so an agent's spend and workload sit in the same picture as everyone else's. You catch capacity clashes between agents and people in time to act, instead of discovering them at quarter-end.
The reason this matters is in the data. In Tempo's 2026 State of SPM report, teams that plan and govern dynamically delivered measurable ROI on 81% of their projects, against 45% for teams that plan statically.
The same research puts the cost of strategic drift, the slow slide between plan and reality, at $260 million a year for every $880 million of strategic spend, roughly 30 cents on every strategic dollar.
Add a workforce of agents that draw budget continuously and never file a status update, and that drift builds faster and surfaces later. Portfolio governance catches it early. Model governance was not designed to.
Push the same logic one step further, and even the compute an agent consumes gets tied to the outcome it was funded for, the way labor cost is today.
Where this is heading: Continuous governance of human and agent work
AI governance and portfolio governance are converging into one discipline, and that discipline is moving off the quarterly calendar.
The quarterly review made sense when work moved at human speed. It doesn’t survive a workforce that includes agents acting around the clock.
Governance has to cover human and agent work in a single view because splitting them rebuilds the silos you spent a decade removing. None of this asks for a new category of tool. It is the governance you already run for human work, extended to every worker in the portfolio and the speed the work now moves.
Start with the half you can act on today: Map your AI agents into the same portfolio view you use for people, what they consume and what they deliver.
The organizations that do it first will govern people and agents as one workforce, in real time, while everyone else is still reconciling after the quarter closes. For a closer look at what a workforce of AI agents changes for the PMO, read how AI enables Strategic Portfolio Management.
Frequently asked questions
1. What is the difference between an AI governance framework and an AI risk management framework?
An AI governance framework is the rulebook; An AI risk management framework is one process inside it. Governance sets who is accountable, and how AI aligns with the business. Risk management is the operational practice of finding and mitigating specific risks like bias or data leakage, the work the NIST AI RMF is built for. You need both: Governance decides the policy, risk management carries it out.
2. What are the main AI governance frameworks?
Most enterprises build on the NIST AI Risk Management Framework, voluntary US guidance that many use as their operating backbone. On top of it they layer the EU AI Act where binding law applies, or ISO/IEC 42001 where they need certifiable proof.
3. Does the EU AI Act apply to AI agents that make resourcing or staffing decisions?
Potentially yes. The EU AI Act classifies AI used for "employment, worker management and access to self-employment" as high-risk, which carries obligations for risk management, logging, and human oversight. If an agent allocates people to work or shapes who gets assigned what, it can fall in that category.
4. How is governing an AI agent different from governing an AI model?
Governing a model asks whether its output is accurate and secure, and whether it treats people fairly. Governing an agent is about whether the action it took was allowed, and whether the work it did was worth the capacity and budget it consumed. A model produces an output a human acts on, while an agent acts on its own, so governance has to cover its decisions and its place in the portfolio.
