Why the First Step in Evaluating an AI Agent Is Finding Its Mistakes by Hand

In the last article, we looked at what an agent’s trace is. Now let’s use the trace to evaluate the output it produced for James.

James is the high-net-worth client we’ve been following through this series. He sold his software company and plans to bring $5 million into the bank. Of that, $4 million comes from the business sale and $1 million from crypto funds.

The onboarding system uses four specialist agents to check different parts of his case. Their findings go to the Risk Engine, which applies the bank’s risk rules. It assigns James a risk level and flags any gaps that still need attention from the compliance officer.

The Case Summary Agent then takes the Risk Engine’s output and turns it into plain English for the compliance officer. It is the only agent in this system that uses an AI model, so it is the only agent we are going to evaluate.

Let’s first see what the Case Summary Agent produced.

1. OPENING LINE
James Whitmore – High-risk client.

2. WHAT HAS BEEN VERIFIED

- Identity verification is complete.
- No sanctions match was found.
- Wealth documents and bank statements have been provided and are on file.
- The business-sale transaction is supported by the relevant documents and
  the corporate registry.

3. WHAT NEEDS REVIEW

- Political-exposed person (PEP) status: James Whitmore has been identified
  as a PEP, which raises a higher risk of corruption or undue influence.
  Enhanced due-diligence is required to understand the nature of his public
  role and any related connections.

- Crypto-derived funds: He has declared cryptocurrency holdings and supplied
  exchange records, but the source of those crypto funds has not been
  established. Without a clear origin, there is a risk of money-laundering or
  financing illicit activity.

4. WHERE TO FOCUS

- PEP due-diligence: Obtain a detailed description of his current and former
  public positions, any family or close-associate relationships, and the
  purpose of the new account. Request supporting documentation (e.g., official
  appointment letters, public disclosures) and perform a risk-based assessment
  of the political exposure.

- Crypto fund origin: Ask for the full transaction history from the exchange,
  including inbound and outbound transfers, wallet addresses, and any KYC
  information the exchange holds on the counterparties. Verify whether the
  crypto was acquired through legitimate means (e.g., mining, sale of assets,
  investment) and document the source-of-wealth narrative.

These steps will address the outstanding high-risk factors before the case
can move forward.

To know whether the Case Summary Agent did what it was supposed to do, we need its map and trace.

The map tells us what we expected the agent to do. The trace shows what it actually received and produced when it ran.

Here is its map that we built in a previous article.

Rows three and four say that the only input the Case Summary Agent gets is from the Risk Engine. So there are two ways the summary can go wrong. The agent writes something the Risk Engine never gave it, or the agent leaves out something it did. Both are visible when you put the two side by side.

So we are going to compare the Risk Engine’s output with what the Case Summary Agent produced.

The Risk Engine’s output is on the left, and the Case Summary Agent’s output is on the right. The highlighted text shows what the agent added, and none of it came from the Risk Engine.

One thing worth noticing is not visible here at all. The Risk Engine’s reasoning said enhanced due diligence is required under FINTRAC guidelines, and FINTRAC doesn’t appear anywhere in the summary.

Let’s note the issues down in a table.

The numbers in the first column point to the highlighted text in the table above. Issue five has no marker there, because you cannot highlight something that is missing.

When you’re noting issues down like this, don’t worry about making them perfect or grouping similar ones together. That comes later.

Four of these are the agent adding content it wasn’t given. The fifth is the agent dropping content it was.

The first four all show up in the same two sections, What Needs Review and Where to Focus.

To understand why the agent did this, we need to look at its prompt. Here are the two sections where the issues showed up.

Now look at What Needs Review.

The agent was asked to list every item that needs attention, and it did that. But it was also asked to explain why each one matters, and the Risk Engine doesn’t give that to the agent. It just hands over the findings.

So the agent filled in its own explanation. It wrote that politically exposed persons carry a higher risk of corruption, and that unestablished crypto origin brings a risk of money-laundering.

Now zoom in on Where to Focus.

The prompt asks the agent to tell the compliance officer exactly what to do next for each unresolved item. The Risk Engine supplies none of that, so the agent wrote the next steps itself, out of its general AML knowledge.

Which leaves two areas that need to be fixed.

The prompt. It needs to say clearly that the agent’s job is to translate what the Risk Engine gave it, and that it should not add anything of its own. It also needs to carry across everything it was given, including the regulatory reference it dropped. Improving a prompt like this is called prompt engineering, and we’ll look at those techniques in a future article.

The Risk Engine. Today its only job is to take the findings from the four specialist agents and apply the bank’s risk rules. It doesn’t send the next steps to the Case Summary Agent because those next steps have to come from the individual agents. When the Screening Agent confirms that James is a politically exposed person, it should also say what the compliance officer does next. The Wealth and Funds Review Agent is the same. When it can’t establish where the crypto funds came from, it should say what happens next.

For now, let’s not worry about the fixes. We’ll come back to them in a later article.

You might be wondering why we need the map and the trace at all.

All we did was put two pieces of text side by side and read them, and that hardly requires a framework.

That’s true for this example. We have one AI agent with a single input and a single output.

A real banking system could have several AI agents, each using different information, tools, databases, and models.

Now imagine the Screening Agent used an AI model and returned the wrong PEP status. The Risk Engine would take that finding and apply its rules. The Case Summary Agent would then turn the result into the summary the compliance officer reads.

The summary could look perfectly fine because it matches what the Case Summary Agent was given.

To find the real problem, you have to work backwards through the trace until you reach the agent that first went wrong. The trace lets you do that and the map tells you what that agent was supposed to do.

That’s why you need both.

We have only evaluated the agent on one case, and that case is James.

Five issues from one case tell you what can go wrong. They don’t tell you how often it happens, and until you know that, you can’t decide what to fix first.

To properly evaluate the Case Summary Agent, we need to run it against many more cases like James’s. Generating those cases is what we’ll look at in the next article.