How Finance Is Rethinking Autonomy for AI Agents

Arthur Azizov argues that as AI agents take on more autonomous work, the real question for finance is no longer how many jobs they can replace, but how much unsupervised authority they can safely be given, and that the right level of autonomy should track how reversible an agent’s actions are.

If you’ve followed the conversation around AI agents over the past couple of years, you’ve probably seen the same prediction repeated over and over that they’re going to replace hundreds or even thousands of jobs. Their ability to analyse documents, write code and automate increasingly complex workflows has dominated headlines for the past couple years. It made people believe that the biggest question is how much work they can take off people’s hands, but that is actually not the main issue, at least anymore.

As AI agents become more capable, it is more important to see whether they can be trusted to do it consistently. Obviously, replacing ten junior employees means little if the system makes costly mistakes that a human team would have caught. And finance is one of the first industries being forced to solve this problem as even a single mistake there can cause huge financial loss.

Finance hits the wall first

If we look closer, we will see that the root of the problem lies in the nature of AI tools as large language models are built on probability. Rather than knowing whether an answer is correct, they predict the most likely next words based on the prompt they receive. That means there is always a chance they will produce a confident but incorrect response, no matter how detailed or carefully written the prompt is.

An incorrect answer to “How do I cook lasagna?” leaves users a little annoyed and doesn’t cause any real harm. The failure in a payment system, in turn, could send customers’ money to the wrong account. Even if the chance of such an outcome were only 1%, it would still be like trusting funds to a casino, few people would be comfortable trusting their finances to it.

And in practice, the risk is 95% of executives said have experienced at least one problematic AI incident, and most of the time (77%), the damage from an AI incident is a direct financial loss to the business.

Even some of AI’s strongest supporters have faced it. About two years ago, the fintech startup Klarna announced that a single AI-assistant would replace up to 700 employees. However, a year later, its CEO said the firm had “gone too far,” and the company quietly restored many positions. The AI-bot was quite bad at managing customers’ problems and too often gave incorrect answers about fees and refunds. In banking, this was unacceptable, and the managers knew it.

Making an autonomous agent less autonomous

Due to all the mistakes agents make, there is a popular idea to treat them as junior employees who, surprisingly, received unusually broad access. If it were so, managing agents would be quite easy. But junior hires can easily stop making the same mistakes, do not run ten thousand operations per minute and, what’s more, do not silently exceed their access rights. Agents are doing all of it.

Also, AI agents have trouble with memory and simply forget what they were doing. If it takes 12 steps to finish the task, a small hallucination at the beginning may lead to an actual catastrophe at the end. That is why the biggest institutions are trying to make their AI agents even less autonomous instead. Morgan Stanley, for instance, built an internal system that matched hundreds of thousands of attributes in the company’s data, and, importantly, a person had to sign off on every call.

Eventually this approach even led to cutting down the workload. The system remembered the same mistakes, and responses to them were made into a fixed rule. The same strategy was used when the bank opened its platform to clients’ external agents, which were run through a permissioned protocol, and today the pilot project is working successfully.

Reversibility is key

Today, I think almost everyone would agree that AI decisions should be overseen by people. Even regulators have reached the same conclusion. The EU now requires high-risk systems, such as credit scoring, to be built with effective human oversight in mind, though under the EU’s 2026 “Digital Omnibus” regulation these high-risk obligations (including human oversight) are deferred and will apply from 2 December 2027. If not, a company may face a fine of up to 7% of its global turnover for the most serious violations under the Act.

But how exactly agents should be managed and what that oversight should actually look like is still unresolved. Should employees check every action or only set the rules above the system settings? Or can they be just a supervisor who steps in when something breaks?

Obviously, there is no answer that would work for everything and everyone. However, a useful starting point here is to ask whether an agent’s actions can be reversed. Falsely flagged anomalies are easy to correct and inexpensive to review, so agents performing such tasks may require little or even no supervision. Settled payments, however, cannot be undone as easily, so in cases like these, the final decision should remain with a human instead.

The autonomy granted to an agent should never exceed reversibility, and finance was the first to understand this. Today, it has an opportunity to set the standards for other industries that eventually may also face the failures of AI agents.