Let’s say you’ve just finished building a banking AI agent that onboards high-net-worth clients.
You’ve spent weeks rigorously testing it, and the pilot confirms that the agent handles complex cases correctly and cuts onboarding time significantly.
Leadership is eager to roll it out.
You send the agent to the review team for final approval before it goes live.
A few days later, the response comes back saying that the agent cannot go live.
You’re shocked because the agent works exactly as expected.
So why wasn’t it approved?
To answer that, you first need to understand who this review team is and what they actually do.
Inside every bank is a team responsible for reviewing AI agents before they are allowed to go live. They sit between the developers who build the agents and the regulators who audit the bank.
In many banks, this team is called the Model Risk team.
Their job is to make sure every AI decision can be explained if a regulator asks about it.
Imagine a regulator walks into the bank six months later and asks, “Show me exactly how this high-net-worth client was onboarded.”
The bank has to answer that question clearly. That means answering questions such as:
- Why was this decision made?
- Which version of the AI model made that decision?
- What evidence did the model use?
You’re probably wondering what could be missing. After all, observability tools like LangSmith already capture every step the agent took.
But the model risk team is asking something else entirely.
Let me show you what they are looking for, using the onboarding agentic AI system I built as a working prototype for this series. If you want to understand how the complete system works, the full walkthrough is here.
Take a client like James, who sold his software company for $18 million and wants to transfer $5 million into a new Canadian bank account.
The onboarding AI system processes James’s application using several specialist agents.
One verifies his identity.
Another screens him against sanctions and PEP registries.
Others review his wealth, source of funds, and business history.
Together, those agents assess James’s application and prepare a complete case for the compliance officer.
The case shows how risky James is as a client, what has already been verified, and what still needs the officer’s attention.
When the system finishes processing James’s application, it leaves behind a complete record of every step it took
Here is what the agent trace looks like in LangSmith, the tool we use to monitor the onboarding system.
You can see which agents ran, how long each one took, and what each agent received and returned. The four specialist agents finish in a few milliseconds. Almost all the run time is the single AI model call at the end.
Now, let’s look at the case summary agent’s output for James, copied exactly from the trace.
1. OPENING LINE James Whitmore – High-risk client. 2. WHAT HAS BEEN VERIFIED - Identity verification is complete. - No sanctions match was found. - Wealth documents and bank statements have been provided and are on file. - The business-sale transaction is supported by the relevant documents and the corporate registry. 3. WHAT NEEDS REVIEW - Political-exposed person (PEP) status: James Whitmore has been identified as a PEP, which raises a higher risk of corruption or undue influence. Enhanced due-diligence is required to understand the nature of his public role and any related connections. - Crypto-derived funds: He has declared cryptocurrency holdings and supplied exchange records, but the source of those crypto funds has not been established. Without a clear origin, there is a risk of money-laundering or financing illicit activity. 4. WHERE TO FOCUS - PEP due-diligence: Obtain a detailed description of his current and former public positions, any family or close-associate relationships, and the purpose of the new account. Request supporting documentation (e.g., official appointment letters, public disclosures) and perform a risk-based assessment of the political exposure. - Crypto fund origin: Ask for the full transaction history from the exchange, including inbound and outbound transfers, wallet addresses, and any KYC information the exchange holds on the counterparties. Verify whether the crypto was acquired through legitimate means (e.g., mining, sale of assets, investment) and document the source-of-wealth narrative. These steps will address the outstanding high-risk factors before the case can move forward.
The summary tells the bank what has already been verified, what still needs attention, and where the compliance officer should focus. For James, those areas are his PEP status and the source of his crypto funds.
At first glance, it looks like the agent trace captured everything.
But the model risk team is not looking at what the agents decided. What they want to know is what produced each decision, and whether the bank can recreate it months or even years later.
That means answering questions like:
- Which version of the AI model generated the summary?
- Which prompt version was active?
- Which documents and external data sources were used?
- Which API responses contributed to the decision?
LangSmith does not capture all of that because much of it lives in external systems.
Now imagine if each of the agents above also used an AI model instead of following fixed rules.
The identity agent might use one model, the screening agent another, and the wealth agent a third.
Each would run on a different model version, follow its own prompt, and call a different set of external services.
Every one of those inputs needs to be recorded with its version.
The reason for doing that is simple.
If a regulator asks six months later how James was onboarded, the bank must be able to reproduce that decision using the exact models, prompts, documents, and data sources that were active at the time.
That is what the model risk team needs before they approve the system to go live.
Now you know why the review team rejected the agent.
It wasn’t because the AI made the wrong decision. The tests passed. The pilot was successful. The agent worked exactly as expected.
The problem was that the bank couldn’t fully explain how those decisions were produced. The agent recorded what it decided, but it didn’t capture everything the model risk team needed to recreate those decisions months or even years later.
That’s the difference between building an AI agent and deploying one inside a bank.
An engineering trace helps developers understand how the system ran. A regulatory record helps the bank explain every decision to a regulator using the exact models, prompts, documents, and data sources that were active at the time.
Until both records exist, the Model Risk team has only one answer.
The agent works. But it still can’t go live.