Borrowed Autonomy: Why Enterprise AI Agents Cannot Simply Be Switched On
There is a difference between a system that works in a demonstration and one that does not destroy value when no one is watching. With AI agents, that difference is measured in modified records, processed payments, and decisions made without human intervention. The problem is not technological. It is one of governance.
An analysis published in the Forbes Technology Council by a senior manager of generative AI at Amazon Web Services places the debate exactly where it belongs: not in model capabilities, but in the oversight architecture surrounding the agent. The operational question it poses is brutally simple. Before deploying an agentic system, you need to know whether your organization can detect its mistakes after they occur, and whether that detection can change what the system does. If the answer to either question is no, the decision to grant autonomy is premature, regardless of the benchmarks the model has passed.
That may sound conservative in a market where Grand View Research estimates that the global agentic AI segment will reach 22.9 billion dollars in 2026 and projects 224.5 billion for 2033. But market size is not a safety argument. It is an argument for urgency in building correctly.
The Refund Model That Reveals the Underlying Problem
The analysis uses an example that seems mundane but proves to be precise: an agent that manages customer refunds. The first distinction it draws is between drafting a response and issuing the refund. The former produces text that a human can review. The latter modifies the state of a bank account.
That distinction is not semantic. It defines the type of error that is possible, its reversibility, and its cost of detection. A poorly worded text generates friction. An incorrect payment generates financial loss, regulatory exposure, and, depending on the sector, legal consequences. The speed at which an agent can generate errors is exactly proportional to the speed at which it can operate.
The proposed approach is to structure human approval around prior definitions: what cases the agent can handle, and what evidence it must have before acting. Incomplete order records, ambiguous policies, or unspecified conditions must trigger an automatic escalation, not an improvised response from the model. This transforms human oversight from a vague concept into an operational control with explicit criteria.
What makes this architecture interesting is what it reveals about the most frequent mistake in agentic implementations: assuming that a high completion rate equals correct outcomes. A system can complete 95% of cases and still produce systematic errors in the remaining 5% without anyone knowing, because no one is looking at the apparently successful cases. The recommendation is to also sample resolutions with no incidents, not only escalations. Otherwise, the agent decides which parts of its own work receive scrutiny, which is exactly the opposite of oversight.
A 2026 study that evaluated 15 models across two different benchmarks found that improvements in capability came accompanied by modest gains in reliability. The data matters because it breaks the most common intuition: that a more powerful model is proportionally safer to deploy. Power and consistency do not scale at the same speed.
The Economics That Reports Usually Hide
The analysis proposes a hypothetical calculation with 1,000 refund cases: 850 resolved by the agent without human correction, 100 resolved by people, and 50 unresolved. Software and infrastructure cost 400 dollars; validation, review, and recovery cost 1,600 dollars. The result: 950 verified cases for 2,000 dollars, approximately 2.11 dollars per case. The calculation excludes initial implementation costs and the consequences of undetected errors.
The arithmetic seems favorable. But the usual way of presenting that same case would be to report 95% completion and attribute the savings to the agent. What that report omits is the human labor contribution, which in the example represents 80% of the total process cost. If that cost is absorbed by another team or budget line, the decision to scale may be made on an artificially reduced cost basis.
There is a 2024 study, "AI Agents That Matter," which argues exactly this: most agent evaluation practices focus too heavily on raw accuracy and ignore cost. For a deployment decision, the correct calculation must include the time that people spend reviewing, correcting, and completing the agent's work. That time may be invisible in the engineering budget if the operations team absorbs the exceptions without reporting them as a cost associated with the system.
The problem is amplified when scaled. Easy cases, which are the first to be automated, generate the most favorable economics. Extending autonomy to disputed transactions or complex cases requires a separate evaluation, because the economics of simple cases do not predict the time the difficult ones will consume. An agent that processes 850 standard refunds at 0.47 dollars each can generate exceptions that cost 60 dollars in human review per case. The aggregate completion rate continues to look good. The actual operating margin does not.
There is an additional cost that the analysis mentions and that tends to disappear from business models: the review queue needs an owner. Reviewers need sufficient evidence to understand what happened and a way to complete the case without repeating all the prior work. If that capability is not part of the deployment plan, cases wait for approval while the agent continues operating, and the customer waits equally, regardless of how quickly the system generated its initial response.
The Chain That Converts Errors Into Constraints
On October 27, 2026, IBM announced updates to its watsonx Orchestrate platform that include user simulation, deterministic evaluation, production failure investigation, test case generation, and an "AgentOps Agent" designed to improve agent quality before its promotion to production. The product direction is exactly what the Forbes analysis describes from the governance side: connecting what happens in production with system behavior before an expansion of authority is authorized.
That architecture — testing before deployment, plus observation in production, plus updating constraints — is the cycle that turns an agent into something a company can manage rather than simply monitor. The difference is that monitoring detects failures. Managing converts them into policy adjustments, authorization limits, or new test cases that prevent recurrence.
The approach proposed in the analysis has a concrete sequence: start with proposed actions, move to human approval, and allow execution within defined limits once results justify it. Some cases may require approval permanently. A policy change, a growing backlog, or errors discovered after the fact must trigger a review of permissions already granted, not a review of future permissions.
The last step in that chain is the one most easily omitted in real implementations: asking the business owner of the process to go through the unresolved cases and the errors detected during the review, and asking what changed as a result of that analysis. The next decision about autonomy should depend on whether those problems were resolved and on whether the team managing the exceptions has the capacity to absorb additional work. If there is no answer to either of those two questions, expanding the agent's authority is a gamble, not a decision.
Before Granting Permissions, Define Who Absorbs the Cost of Error
The pattern that emerges from this analysis is not new in the history of enterprise automation. Every technological wave, from ERP to RPA, went through a phase in which the promise of efficiency was evaluated on the cases the system handled well, and the real cost appeared in the cases it did not. The difference with language agents is that their flexibility — which is precisely what makes them attractive — also makes their failures less predictable than those of a deterministic rule.
An agent operating on a natural-language instruction can vary its behavior with context, with ambiguous data, with model updates, or with tools that respond differently depending on the state of the system. That means validating it against a finite benchmark in pre-production does not guarantee consistency in production. It guarantees that it passed that benchmark.
The question any operations manager should ask before approving an expansion of an agent's authority is not how many cases it completed. It is how many of the cases it completed were reviewed by someone with the ability to detect an error, and what the organization did with the errors it found. If no one reviewed the successful cases, if the errors found did not modify the system's rules, or if the review team has no visibility into the agent's reasoning, the completion rate does not measure what it appears to measure.
Agentic AI markets will grow with or without governance. What will determine which organizations scale with control and which accumulate silent liabilities is exactly this: whether the cost of error is visible in the correct budget, whether reviewers have the tools to complete the work the agent did not finish, and whether someone is responsible for converting each failure into a constraint that prevents the next one. Autonomy is not granted. It is built with operational evidence, and that process takes longer than the deployment schedule usually admits.
