Essay

Why fraud models should only propose. By Amir Shachar.

Read the essay
Skip to content
Riskernel

Blog · Field notes

Backtest, Shadow Test or Champion/Challenger: How to Evaluate a Fraud Model

Short answer: A backtest replays history and shows whether a model could have worked. A shadow test scores live traffic with no decisions attached and shows how the model behaves now. Champion/challenger lets the new model take real decisions on a share of traffic. Use them in that order, and trust none of them alone.

A fraud model can look right on paper and still fail the day it meets live traffic. Three evaluation methods exist because they answer different questions. I led fraud data science at NICE Actimize and now run Riskernel, which runs free shadow tests, so my examples come from those.

Backtest: what it proves and how leakage inflates it

A backtest replays past events through a model and scores the output against labels you already have. It proves the model could have separated fraud from good traffic on that history. It says less about live behavior, because history is easy to leak into.

Leakage means the test saw something the live system would not have. A random split lets the model train on events that happened after the ones it is tested on, and a feature computed with information that arrives later looks predictive in the notebook but has nothing to work with at decision time. Labels lag too. Late chargebacks make recent fraud look like good traffic, and if labels exist only for events someone reviewed, the backtest scores the model against your past decisions as much as against the truth.

The honest version splits by time and measures what a team would feel: how much fraud the model catches inside the number of alerts the team can actually review. The same trap shows up with rare fraud, which extreme class imbalance covers.

Shadow test: live traffic with no decision impact

A shadow test scores a copy of live or mirrored traffic in parallel while your current system keeps deciding. Nothing sits in the decision path. It proves the model works on your traffic as it is now, with your customer mix, missing fields, peaks and edge cases. You can lay the score distribution against what your current system did, check whether the reasons are usable (Riskernel returns SHAP explanations with each score), and read the events where the two disagree.

It hides outcomes. The shadow model decides nothing, so you never see what would have happened if it had blocked a payment, how customers would have reacted, or how fraudsters would have adapted. Labels also arrive late, so a short run shows where the two systems agree and disagree before it shows who was right. The guide to shadow testing a fraud vendor covers what to compare, and our free 14-day shadow test follows it.

Champion/challenger: when the new model takes real decisions

In champion/challenger the current model, the champion, keeps most decisions and the new one, the challenger, decides a share of real traffic. It proves what the other two cannot: real outcomes, including fraud stopped, good customers declined and the effect on the review queue, measured over the same period and flow.

The cost is exposure. A weak challenger makes weak decisions on real customers. So run it after a backtest and a shadow test, start with a small slice, agree on the number you want to move, assign events or accounts to each side at random so the groups are comparable, and keep a way to switch back.

Which method at which stage

Use them in this order. Each one removes a risk the previous one could not see.

StageMethodWhat it provesWhat it can hide
No live access yetBacktestThe model separates fraud from good traffic on historyLeakage, late labels, a past that no longer matches today
Live traffic available, no authority to decideShadow testBehavior on your real traffic, with reasons and disagreementsReal outcomes, customer reaction, fraudster adaptation
Shadow results agreed, ready to actChampion/challengerReal outcomes of the new model's decisionsReal loss and customer friction if the challenger is weak

If you are choosing between vendors, Sardine vs Unit21 and the vendor comparison show how the same tests apply.

What a day-14 shadow report should contain

The sample report shows the shape, and every figure in it is synthetic. A useful day-14 report states one flow and one agreed number, such as the share of payouts sent to manual review. It lays the score distribution against what your current system did, band by band, and lists the top reasons behind the highest scores. Its findings separate observation from hypothesis, each with its evidence and a first control to try. As the sample puts it, a high score is a reason to look, not a verdict.

The report comes with the raw scores, so your team can verify every number independently. You keep the report whether or not you work with us again.

Questions teams ask

Is a backtest enough to approve a fraud model?

No. A backtest shows the model could have worked on history, and leakage or late labels can make it look better than it will be live. Add a shadow test on live traffic before the model decides anything.

What is the difference between a shadow test and champion/challenger?

In a shadow test the new model scores traffic but decides nothing. In champion/challenger it makes real decisions on a share of traffic, so you see real outcomes and also carry real risk.

How long should a shadow test run?

Long enough to see quiet periods, peaks and some analyst-reviewed outcomes, which usually means one to two weeks of real traffic. Ours runs 14 days on one flow.

Do I need to change production to run a shadow test?

No. The candidate scores a copy of the flow, historical or mirrored, and your current system keeps making the live decision.

See the shadow method on your own flow

Tell us the flow and we score a copy of it beside your current system. The free 14-day shadow test ends with the report and the raw scores, and you keep both.