For years my job was to put fraud models into production at large banks. The hard part was never getting a model to score well on a test set. The hard part was the meeting afterwards, when someone from model risk asked a simple question: "Show me why it did that, and show me it will do the same thing tomorrow."
That question is about to get much harder to answer. Generative AI is moving into fraud and payments operations quickly. It writes rules, drafts alert narratives, maps new data feeds and, more and more, takes actions: it freezes an account, holds a payout, closes an alert. The way most teams are wiring it in has a common shape, and I think the shape is wrong for this domain. Not wrong for every domain. Wrong for money.
The pattern most teams are building
Call it agent-in-the-loop. A model, often an agent with tools, looks at a case and acts. Around it sit the safety measures: a person who reviews a sample of decisions, or signs off on certain classes of action; detailed logs of what the agent saw and did; a kill switch if something goes badly; and increasingly a second model that grades the first one’s work.
I want to be fair to this pattern, because the people building it are not careless. Human review catches real mistakes. Logs matter. A kill switch is better than no kill switch. A second model can spot things a rule never would. In customer support, in research, in drafting, this is a reasonable design. If the agent gets something wrong, a person fixes it and little is lost.
The trouble is the order of operations. The model decides, the action runs, and the checking happens afterwards, or in parallel, or on a sample. Every one of those safety measures is about finding out. None of them is about preventing.
Why the order matters in fraud and payments
Three things make fraud different from most places agents are being deployed.
First, actions are hard to reverse. A payout that left is gone. A legitimate customer whose account was frozen on a Friday night does not experience the Monday correction as a fix; they experience it as the weekend their rent bounced. "We reviewed it later and reversed it" is a real cost, not a footnote.
Second, the adversary reads your system. Fraud is the one place where someone is actively probing your decision logic for the cheapest way through. A model that reads free text (a memo field, an invoice, a chat message) is a model that an attacker can write to. A reviewer who checks a sample after the fact is reviewing the cases the attacker chose to let you see.
Third, you will be asked to explain. Regulators and auditors do not want to know what the system usually does. They want to know what it did on this account, on this date, and why. "The model judged it, and a second model agreed" is not an answer a model-risk team can defend. It is two opinions where they asked for a reason.
Put those together and the agent-in-the-loop order looks backwards. You want the checking to happen before anything executes, not after.
Models propose. A validator decides what runs.
This is the idea behind Algogens, which I first set out in a 2024 paper (arXiv:2403.01426) and which Riskernel’s engine is built on. The short version: the accuracy of algorithms plus the intelligence of generative AI, with each doing only the part it is good at. The method Riskernel’s engine uses is patent pending (US provisional application 64/197,364, filed in November 2026).
The generative model is kept out of the decision path. It works offline, where it does what it is genuinely good at: proposing. It suggests a new feature, a new rule, a mapping for a data feed nobody has seen before. That is creative work, and generative models are good at it.
Every proposal then goes through a set of deterministic validators. Deterministic means the same input always gets the same verdict; there is no model inside the check and no sampling. There are two sets, and they check different things. The first set checks the fraud rules and features a model generates, and it is the one in the paper. Does this feature use information that would not have been available at decision time? Is this pattern spurious? Is it acting as a proxy for a protected attribute? And how many past decisions would this rule change if we turned it on? The second set checks the actions an AI proposes to take, such as freezing an account: schema integrity, telemetry grounding, regulatory policy consistency and blast-radius solvability. Neither set uses a model to judge.
On generated rules, that last check is the one I find operators understand immediately. Before a rule is admitted, it is replayed against history, and the share of past decisions it would have changed is measured against a fixed budget. A rule that would have flipped too many decisions is not simply accepted or refused.
Repair within bounds, or reject
This is the part that separates the approach from a plain approve or deny gate. When a proposal fails, there are two outcomes, and only two.
It can be repaired within bounds. If a proposed rule wants to block, but blocking would change too many past decisions, it is cut back to the strongest milder action that fits the budget: a hold or a review instead of a block. The repair itself is deterministic, so the same failed proposal always gets the same repair.
Or it is rejected. If no action above "allow" fits the budget, the proposal is dropped. A rule whose false-positive rate is too high is dropped. There is no third path where the model argues its case, and no path where a failed proposal runs "just this once".
Only proposals that pass are admitted. Nothing that failed runs as proposed.
What this buys a CRO
A chief risk officer, in my experience, wants to know three things: what can this system do, what has it done, and what is the worst it can do.
With Algogens the first answer is short. The system can only take actions from a fixed list, and only through rules and features that were checked before they were loaded. The model’s creativity is real, but it lives upstream of a gate it cannot talk its way through.
The second answer is a log, but a different kind of log. Not just the agent’s output, but every proposal, every validator verdict and every repair. You can see the rules that were rejected, not only the ones that won.
The third answer is a bound. Each admitted rule’s effect on past decisions was measured and capped at admission. That is a number someone chose in advance, written down, not a hope.
What it buys a regulator
Replay. A reviewer can take a decision from last quarter and walk the path step by step: which checks it passed, whether it was repaired, which action followed.
That is the difference between showing an outcome and showing a decision path. Agent-in-the-loop systems can usually show the outcome and the agent’s account of itself. An Algogen can show the path.
What it does not buy
I would rather say this myself than have someone find it. A validator only checks what it was written to check. The paper reports synthetic tests only, and I would not claim more for the design than that. A deterministic gate does not make a system correct. It makes the system’s behavior bounded, inspectable and repeatable, which is the precondition for making it correct and for proving it.
That is also why the architecture matters more than any single model. Models will keep getting better, and they will keep being wrong in new ways. The question is not whether your model is good enough to trust. It is whether your system is built so that it does not need to be.
Models should propose. Something you can read, test and replay should decide what runs.
Amir Shachar