Why Evaluating AI Agents is So Hard in Banking

You already know how to test software. You feed it an input, check the output against the answer you expected, and know right away whether it passed or failed.

AI agents are different. Run the same input twice and you can get two different answers, both of them perfectly reasonable.

So what do you compare the output against?

The easiest way to answer that question is to go back to James’s onboarding.

Here’s the system we’ve been building throughout this series.

The specialist agents have already done their work and passed their findings to the bank’s risk engine.

The risk engine applies the bank’s risk rules and produces its output as JSON, a collection of codes and fields that isn’t easy for a compliance officer to read.

A case summary agent translates that JSON into plain English for the compliance officer.

Only one agent in this system actually uses an AI model: the case summary agent. 

The specialist agents and the risk engine all run on rules the bank has written down. Give them the same input and they return the same output every time. That makes them straightforward to test. 

The case summary agent is the only part of this system that uses an AI model. In this prototype, that’s GPT-OSS 120B through Groq.

Here is the prompt the case summary agent uses.

The prompt is detailed. It tells the model what to write, how to structure it, and what to leave out.

Now let me run the onboarding system twice on James’s case.

Here are both outputs.

The first summary tells her to obtain documentation from James regarding the source of the crypto funds. The second tells her to investigate the crypto exchange’s records.

The regulatory references change too. The first summary refers to FINTRAC guidelines twice, and the second never mentions them at all, even though the Risk Engine explicitly included them.

Nothing changed between those two runs. The same prompt produced two different summaries. 

So which one is correct?

We can’t answer that question yet. Nobody has ever written down what a correct summary looks like.

And if we can’t define “correct,” neither can the AI.

So before we can evaluate the model, we first have to define what a good answer looks like. 

For James, the risk engine produced this output. It typically arrives as messy JSON, so I’ve cleaned it up here to make it easier to read. 

From that output, the bank can define the evaluation criteria. 

A correct summary must:

• State the risk signal and explain why it was assigned
• Include every item the risk engine flagged for review
• Preserve any regulatory references the risk engine supplied
• Tell the compliance officer what to do next in clear, actionable language
• Avoid exposing internal system codes

Those criteria don’t come from the team building the agent. They come from the compliance officers who already write these summaries today.

Now go back to the two outputs.

The second summary removes the FINTRAC reference even though the Risk Engine supplied it, so it fails the evaluation while the first summary passes.

That’s the real value of an AI eval. 

You aren’t simply saying one answer reads better than another.

You’re measuring every answer against the same agreed standard, which tells you exactly what broke and what needs fixing.

But James is just one client.

A real bank processes hundreds of different clients. Some declare crypto assets, others are politically exposed persons, and many have several sources of wealth that all need verifying.

So the bank builds a large collection of client cases covering that range and runs the agent against every one of them. 

Some summaries meet the criteria and others don’t. Each summary that fails the evaluation tells the team exactly what needs fixing.

The problem is that no compliance team can manually review thousands of summaries every time the agent changes. So the bank automates the entire evaluation process. 

Every time the prompt or the model changes, the agent runs against the same collection of client cases. The system checks whether each summary meets the criteria and reports anything that breaks before the agent goes live.

So why does all of this matter so much in banking?

Go back to the evaluation criteria for a moment.

One line says the summary must include every item the Risk Engine flagged for review.

Now imagine that line was never there.

The Risk Engine correctly flags that the origin of James’s crypto funds still needs to be reviewed, but the case summary agent leaves that point out. 

The compliance officer never sees it. She investigates the PEP finding, resolves it, and approves James’s application. As far as she’s concerned, every issue has been addressed. 

Years later, regulators review the case and ask why the bank never looked into the source of James’s crypto funds.

The bank can’t answer that the AI forgot to mention it. From the regulator’s perspective, the decision wasn’t made by the AI. It was made by the bank, and the AI was simply another system the bank chose to use.

The bank may now face millions of dollars in regulatory penalties because it approved James without investigating the source of his crypto funds.

That’s why AI evaluations matter so much in banking. A missing sentence in a summary isn’t just a model error you fix in the next sprint. It’s a compliance failure with real financial consequences.

Those consequences are exactly why AI agents have to be evaluated differently.

Traditional software is easy to test because you already know the correct answer. AI doesn’t give you that certainty. 

Instead, the bank defines what a good answer must contain and checks every summary against those criteria.

That is an AI evaluation.

In banking, it isn’t just another testing exercise. It’s how you know your AI agent is ready to work with real customers