Essay

Why fraud models should only propose. By Amir Shachar.

Read the essay
Skip to content
Riskernel

Free tool · Runs in your browser

Shadow test sample size calculator

How many fraud cases, and how many days of shadow running, it takes to tell a new system from the current one. Enter your volume, fraud rate and detection rate, and the smallest gain that would change your decision.

Everything is worked out in your browser. Nothing you type is sent or stored. The link button only builds an address that carries your inputs.

How this is calculated

The test compares the share of fraud cases each system catches. Write p1 for the current system’s detection rate and p2 = p1 + d for the challenger, where d is the smallest improvement worth detecting. With confidence 1 - a (two-sided) and power w, the number of fraud cases is:

pbar = (p1 + p2) / 2

n = ( z(1 - a/2) x sqrt(2 x pbar x (1 - pbar))
    + z(w)       x sqrt(p1(1 - p1) + p2(1 - p2)) )^2 / (p2 - p1)^2

fraud cases per day      V x r
days of shadow running   ceil( n / (V x r) )
days to a final answer   days of running + label delay

Here z is the standard normal quantile, V is events per day and r is the base fraud rate.

Assumptions: both systems score the same traffic, so each fraud case is a matched observation and the count that matters is fraud cases, not events or days. The formula is the standard two-proportion test and treats the two systems as independent samples. Matched cases are correlated, which lowers the number you need, so the result is a cautious upper bound. Fraud arrives at a steady rate. Label delay is the time before a case is confirmed, for example after a chargeback window, and it comes on top of the running time because the last cases cannot be judged until their labels arrive.

Why count fraud cases and not days

A shadow test with plenty of traffic and few fraud cases proves nothing about detection. The evidence sits in the fraud cases the two systems either catch or miss, so the number of cases sets the length of the test. The table under the result shows how fast that number grows when the gain you want to detect gets smaller.

Frequently asked questions

What is a shadow test?

A second system scores live traffic beside the current one, and its decisions are recorded but not acted on. You compare the two on the same cases before anything changes in production. See the shadow test page for how it runs.

Why does a smaller improvement need so many more cases?

The number of cases grows with the inverse square of the gain. Halving the gain you want to detect roughly quadruples the cases, which the table below the result shows for your inputs.

What should I enter as the smallest improvement worth detecting?

The gain that would make you switch systems, in percentage points of fraud caught. A gain below that does not change your decision, so there is no reason to size the test to find it.

Why does the label delay matter?

Fraud is confirmed late. Until a case has its label, neither system can be scored on it. The final answer arrives after the last fraud case in the sample is confirmed.

Does the calculator send my numbers anywhere?

No. Everything is worked out in your browser. The “Copy link” button puts your inputs in the page address so you can share a result, and nothing else.

Use this calculator on your own page

Paste this snippet into a page you control. The embedded version has no site menu and one credit link back here.

Go deeper