To evaluate an AI agent, its final answer isn’t enough.
You also need to know what information it used, whether it followed the instructions it was given, and where that information came from.
To see all of that, you first need a map.
So what is that map?
An agent map holds four things.
- What the agent is expected to do
- What regulations it has to follow
- What information it uses
- How it accesses that information
Let’s build a map for the onboarding system we’ve been using throughout this series.
The onboarding system is what we built to bring a high net worth client onto a bank’s books.
The orchestrator routes the client’s information to four specialist agents. Their findings go to the risk engine, which applies the bank’s rules and classifies the client as HIGH, MEDIUM, or LOW risk. The case summary agent then translates those findings into plain English for the compliance officer.
Only the case summary agent uses an AI model, so that’s the one we’ll build a map for.
Let’s start with the first part of the map.
What the agent is expected to do
At a high level, the case summary agent translates the risk engine’s output into plain English. Let’s break that requirement down into its individual parts.
- Structure. The summary splits into four parts. An opening line, what has been verified, what needs review, and where to focus.
- Coverage. Every verified item appears, and every item needing review appears. Nothing gets skipped.
- Explanation. Each item needing review comes with a reason why it matters, written in plain English.
- Next steps. Every unresolved item tells the compliance officer what to do next.
- No internal codes. The officer reads English, not PEP_CONFIRMED or WEALTH_SUPPORTED.
- No recommendation. Approve and reject are words the agent never uses.
- Length. Under 250 words.
- Only what it is given. Every statement comes from something the risk engine supplied. The agent adds nothing of its own and softens nothing it received. If the risk engine says a case cannot proceed, the summary can’t turn that into a suggestion.
A case summary agent sounds simple until you write down everything it’s expected to do.
All of those requirements become part of the agent’s prompt.
This prompt can still be improved. Examples of good summaries would make it stronger, and there are prompt engineering techniques that would sharpen it further. We’ll come back to both in a later article. For now, these requirements define what the agent has been asked to deliver.
What regulations it has to follow
An agent doesn’t just follow the instructions in its prompt. It also has to follow the regulations that apply to it, and those regulations shape how the agent is built.
In Canada, banks using AI agents fall under OSFI Guideline E-23. Among other things, it requires AI-assisted output to be explainable and documented. That means every input the case summary agent receives and every summary it produces has to be recorded, so someone who didn’t build the agent can still review its work. Most other countries already have similar requirements in place, and those that don’t are working towards them.
Two more regulatory requirements apply to this agent.
The compliance officer makes the decision, not the agent. The bank remains accountable for every onboarding decision, so nothing the agent writes can pre-empt the compliance officer’s judgement. That is where the rule against using words like approve or reject comes from. It was never a style choice.
Another requirement comes from the PCMLTFA. If a client is flagged in a sanctions match, the summary cannot instruct the compliance officer to ask the client about it, because doing so could amount to tipping off the client, which is an offence under the Act.
All of that applies to an agent whose only job is writing a case summary.
If the identity verification agent also used AI, it would inherit a completely different set of regulations. The Canadian PCMLTFA specifies how a client’s identity can be verified, which methods are acceptable, and when enhanced verification is required. The agent would also have to record which verification method it used for each client and retain that information for years after the relationship ends.
Even within the same onboarding system, each AI agent has its own regulatory requirements.
That’s why every AI agent gets its own map.
What information it uses
Every agent gets its information from somewhere and uses that information to do its job.
Our case summary agent receives a single input from the risk engine and turns it into a plain English summary.
In a real bank, a single agent might receive information from several places at the same time, such as:
- Findings from other agents
- Internal bank systems
- External vendors
- Public registries
- Documents uploaded by the client
All of those information sources need to be captured in the agent’s map.
How it accesses that information
Knowing where the information sits doesn’t mean the agent can reach it, and what gives it that reach is a tool.
A tool is simply a piece of code the agent can call to retrieve information or perform an action.
If the screening agent used AI, one of its tools might check the sanctions provider’s database and return the result to the agent.
Another tool might update the case management system after a review is complete. In that case, the tool performs an action instead of retrieving information.
Our case summary agent doesn’t use any tools because it simply receives the risk engine’s output and turns it into plain English.
One more thing belongs in this part of the map, and that’s the model itself. So does the version of the prompt it runs on.
The model decides which tool to call and when. It’s also what turns the risk engine’s findings into the sentences the compliance officer reads. Our case summary agent runs on gpt-oss-120b, so the model and its version number become part of the map.
Swap that model out and you’ve changed the agent, even though nothing else in the map has moved. Change a single line in the prompt and the same thing happens. That’s why banks record both versions, and evaluate the agent again every time either one changes.
With all four parts written down, the map looks like this.
| Section | Summary |
|---|---|
| What it is expected to do | Turn the risk engine’s output into a plain English case summary. Four fixed sections, under 250 words, no internal codes, no recommendations, and nothing added beyond the risk engine’s findings. |
| What regulations it has to follow | OSFI Guideline E-23. The compliance officer makes the decision. Never tip off a client about a sanctions match. |
| What information it uses | Risk engine output only: risk signal, verified items, items needing review, and reasoning. |
| How it accesses that information | Direct input from the risk engine. No tools. Model: gpt-oss-120b. Prompt: case_summary_v3. |
Four rows is all it takes, and that’s because the case summary agent is a simple agent. The same onboarding system inside a bank might have several AI agents, and every one of them needs its own map.
That sounds like a lot of work, but it isn’t.
Most of this information already existed when you built the first version of the system. During the mapping process, all you’re doing is bringing it together in one place, agent by agent. Once you’ve done that for every AI agent in the system, you’re ready to evaluate them.
Now, when the agent produces an answer, you have something to compare it against.
Instead of asking whether the answer sounds right, you can check whether the agent behaved the way you designed it to.
What the map can’t tell you is whether the agent actually did those things.
When an AI agent produces the wrong answer, you need to see every step that led to it. Which information it actually used, whether it followed the instructions it was given, and where each piece of that information came from.
That record is called a trace, and that’s what we’ll look at in the next article.