Free tool · Runs in your browser
Shadow test sample size calculator
How many fraud cases, and how many days of shadow running, it takes to tell a new system from the current one. Enter your volume, fraud rate and detection rate, and the smallest gain that would change your decision.
Everything is worked out in your browser. Nothing you type is sent or stored. The link button only builds an address that carries your inputs.
How this is calculated
The test compares the share of fraud cases each system catches. Write p1 for the current system’s detection rate and p2 = p1 + d for the challenger, where d is the smallest improvement worth detecting. With confidence 1 - a (two-sided) and power w, the number of fraud cases is:
pbar = (p1 + p2) / 2
n = ( z(1 - a/2) x sqrt(2 x pbar x (1 - pbar))
+ z(w) x sqrt(p1(1 - p1) + p2(1 - p2)) )^2 / (p2 - p1)^2
fraud cases per day V x r
days of shadow running ceil( n / (V x r) )
days to a final answer days of running + label delay
Here z is the standard normal quantile, V is events per day and r is the base fraud rate.
Assumptions: both systems score the same traffic, so each fraud case is a matched observation and the count that matters is fraud cases, not events or days. The formula is the standard two-proportion test and treats the two systems as independent samples. Matched cases are correlated, which lowers the number you need, so the result is a cautious upper bound. Fraud arrives at a steady rate. Label delay is the time before a case is confirmed, for example after a chargeback window, and it comes on top of the running time because the last cases cannot be judged until their labels arrive.
Why count fraud cases and not days
A shadow test with plenty of traffic and few fraud cases proves nothing about detection. The evidence sits in the fraud cases the two systems either catch or miss, so the number of cases sets the length of the test. The table under the result shows how fast that number grows when the gain you want to detect gets smaller.
Frequently asked questions
What is a shadow test?
A second system scores live traffic beside the current one, and its decisions are recorded but not acted on. You compare the two on the same cases before anything changes in production. See the shadow test page for how it runs.
Why does a smaller improvement need so many more cases?
The number of cases grows with the inverse square of the gain. Halving the gain you want to detect roughly quadruples the cases, which the table below the result shows for your inputs.
What should I enter as the smallest improvement worth detecting?
The gain that would make you switch systems, in percentage points of fraud caught. A gain below that does not change your decision, so there is no reason to size the test to find it.
Why does the label delay matter?
Fraud is confirmed late. Until a case has its label, neither system can be scored on it. The final answer arrives after the last fraud case in the sample is confirmed.
Does the calculator send my numbers anywhere?
No. Everything is worked out in your browser. The “Copy link” button puts your inputs in the page address so you can share a result, and nothing else.
Go deeper
- Shadow test: run a second system beside your current one and compare, with no change to production.
- Alert budget calculator: what an operating point does to a day of alerts.