Free tool · Runs in your browser
Base-rate reality check
“99% accurate” sounds like almost every flag is fraud. When fraud is rare, it is the other way round. Enter your event volume, your fraud rate and the model’s rates, and see how many of its flags are fraud and how many are good customers.
Everything is worked out in your browser. Nothing you type is sent or stored. The link button only builds an address that carries your inputs.
How this is calculated
Write N for the number of events, p for the fraud rate, F for the fraud events and G for the good events. The detection rate (also called sensitivity) is the share of fraud the model flags. The false positive rate is the share of good events it flags by mistake.
F = N x p fraud events
G = N - F good events
TP = F x detection rate fraud that is flagged
FN = F - TP fraud that gets through
FP = G x false positive rate good customers flagged
TN = G - FP good customers passed
flagged = TP + FP
precision = TP / (TP + FP)
= p x d / (p x d + (1 - p) x f) (Bayes' rule, d = detection rate, f = false positive rate)
NPV = TN / (TN + FN)
accuracy = (TP + TN) / N
flag nobody = 1 - p accuracy of a model that never flags
In words: the flagged group is a small slice of fraud plus a small slice of a very large group of good events. When good events outnumber fraud a thousand to one, even a small false positive rate produces more good flags than fraud flags. Precision, the share of flags that are fraud, is what an analyst feels. NPV is the share of passed events that are good.
Accuracy mode: when you enter one accuracy figure, the tool uses it as both the detection rate and the share of good events passed, so the false positive rate is 100 minus the figure. That is how a claim such as “99% accurate” is usually read. If a source means something else, switch to the two-rate mode and enter the rates it gives. Assumptions: the rates stay the same at your fraud rate, and every event is labelled correctly.
Why accuracy misleads when fraud is rare
Accuracy counts every correct call, and good events are almost all of the traffic. At a fraud rate of 0.1%, a model that flags nobody is 99.9% accurate. A model that catches 99% of fraud and flags 1% of good customers scores 99.0%, lower than the model that does nothing, and it raises about ten false alerts for every real one. Your team’s workload follows precision at your fraud rate and the number of alerts raised.
Frequently asked questions
Why are so few flags fraud when the model is 99% accurate?
Because good events are far more common. If 1 event in 1,000 is fraud, a 1% false positive rate flags about 10 good events for each fraud event in the traffic, while the model can find at most that one. Most flags are therefore good customers, even though the model is right on 99 events in 100.
What should I ask a vendor who quotes accuracy?
Ask for precision at your fraud rate and at your alert volume: of the events flagged each day, how many were fraud on traffic like yours? Also ask how many alerts a day that means for your team, and how the figures were measured, on what data and over what period.
What does accuracy mode assume?
It treats the single figure as both the detection rate and the share of good events passed. Real models rarely behave that evenly, so use it to read a marketing claim and switch to the two rates once you have measured them.
Does the tool send my numbers anywhere?
No. The figures are worked out in your browser. The “Copy link” button puts your inputs in the page address so you can share a result, and nothing else.
Can I use it in teaching or on my own site?
Yes. Use it in a class, a training session or a briefing, and copy the embed code on this page to put it on your own page. The embedded version keeps one credit link back to this page.
Go deeper
- Alert budget calculator: what those flags do to a review queue.
- Alert budgets, not AUC: why the staffing question needs an alert count.
- The label you get: fraud labels arrive late and only for some events, which shapes every rate above.
- Shadow test: run a second system beside your current one and compare, with no change to production.