How Do You Know Your AI Fraud Investigator Is Right? Testing Agents Against a Known Answer Key

Every fraud and AML operation I know has the same queue problem: more alerts than investigators, and most alerts turn out to be nothing. So it makes sense that teams want AI agents to work those alerts, gathering the KYC file, reading the transaction history, linking accounts and drafting a disposition.

The hard question is not whether an agent can produce a confident write-up. It can. The hard question is whether the write-up is right, and how you would know before you let it near a real case.

The ground-truth problem

To measure an investigator, human or AI, you need cases where the right answer is known. Real case files are poor for this. Labels in fraud and AML are incomplete and late: a closed alert is not proof of innocence, a mule account may be identified months later or never, and the rings you know about are the ones your current controls already caught. Score an agent against historical dispositions and you mostly measure how well it imitates your past decisions, including their blind spots.

Using real customer data to test a new vendor's agent also raises privacy, outsourcing and data-sharing questions before a single test has run.

Why now

Regulators are already asking how firms supervise AI that does real work. FINRA's 2026 Annual Regulatory Oversight Report flags agents acting without human validation and multi-step reasoning that is hard to trace or explain, and expects firms to think about agent-specific controls. Under the SR 11-7 model risk guidance, a model used in decision-making needs validation with effective challenge and outcomes analysis. An AI investigator that recommends closing or escalating alerts is hard to argue out of that scope, and outcomes analysis needs outcomes you can trust.

Public test sets do not solve this either. Researchers have documented benchmark data contamination, where models have seen evaluation data during training, so a published benchmark is a weak basis for an AML go-live decision.

How agent pilots go wrong

• Grading by plausibility. A reviewer reads 50 agent narratives, finds them well written and approves. Fluency is not accuracy.
• Single-alert thinking. The agent dispositions each alert alone and never sees that six "unrelated" alerts share a device, an address and a beneficiary. Rings and mule networks only appear across cases.
• Missed negatives are invisible. If the agent closes an alert that was part of a ring, nothing in a normal pilot tells you.
• Vendor-chosen demos. The agent is shown on the cases it handles well.

Self-check before you trust an AI investigator

• Do you have any cases where the full truth is known? Not "closed as no issue" but known innocent, and known ring members including the ones no alert flagged.
• Does your test include connected cases? At least one ring spread across several alerts, so you can see whether the agent links them.
• Are you measuring misses, not just hits? Count the planted bad actors the agent cleared.
• Does the agent cite evidence you can check? Every claim in a disposition should point to a record in the case file.
• Is the answer key hidden from whoever tunes the agent? If the vendor or your own team can see it, the test will be fitted to it over time.
• Could you rerun the test on a new version next quarter? One pilot is a snapshot. You will need to repeat it every time the model or prompts change.

If you cannot tick most of these, you do not yet know how good the agent is. You know how it reads.

What a useful synthetic back office contains

Synthetic data is only useful for this if it behaves like a real back office. Random transactions with a few flagged rows teach nothing. What matters is structure: customers with plausible histories and KYC files, normal activity that generates false-positive alerts in realistic volume, and planted schemes that look the way real ones do in the data. A mule network should show the money flowing in and fanning out. A synthetic-identity ring should share fragments of identity across accounts that were opened weeks apart. Disputes and complaints should arrive late, as they do in life.

Because the generator planted the schemes, it knows every member of every ring, including the accounts that never triggered an alert. That is the answer key, and it stays hidden from the agent and from whoever tunes it. You can then measure things that are impossible on real data: what share of ring members the agent identified, how many innocent customers it escalated, and whether its stated evidence actually supports its conclusion.

How Riskernel runs this as a fixed-scope assessment

The AI Investigator Proving Ground gives you a synthetic back office: customers, transactions, alerts, KYC files and disputes, with planted fraud rings and mule networks and a hidden answer key. Because it is synthetic, no customer data leaves your organisation.

Your investigator agent works the cases, and you get a scored run against the answer key: what it found, what it missed and where its reasoning went wrong. The answer key reflects typologies I know how to plant, so it is a strong test of known patterns rather than proof against every possible scheme. Scope and fees are agreed on a short call.