High scores cluster on payouts to destinations added shortly before the payout.
In this synthetic set, most events in the top band went to a destination added within the previous hour.
Sample report
Sample report (synthetic data)
What you get at day 14 of a shadow test, laid out the way it arrives. All figures are made up and round on purpose.
1 · Summary Sample · synthetic data
One flow, one number, 14 days.
- Flow
- Instant payouts to newly added debit cards
- Agreed number
- Share of payouts sent to manual review
- Period
- 14 days
- Events scored
- 100,000
- Where it ran
- In parallel. Nothing in the decision path.
2 · Scores against current results Sample · synthetic data
Where the scores fall, and what your system did.
Each event gets a score from 0 to 1. The last column is how many events in that band your current system sent to review.
| Score band | Events scored | Share of events | Current system sent to review |
|---|---|---|---|
| 0.0 to 0.2 | 61,000 | 61.0% | 100 |
| 0.2 to 0.4 | 21,000 | 21.0% | 200 |
| 0.4 to 0.6 | 10,000 | 10.0% | 500 |
| 0.6 to 0.8 | 5,000 | 5.0% | 1,050 |
| 0.8 to 1.0 | 3,000 | 3.0% | 1,450 |
Read first. 5,500 events in the two highest bands were not sent to review by the current system. They are the first events a joint review reads. A high score is a reason to look, not a verdict.
3 · Top reasons Sample · synthetic data
Why the highest scores were high.
Every score ships with its reasons. This is how often each one appeared among the top reasons for events in the two highest bands.
first_time_payee_corridor55%payout_velocity_5m_spike45%new_beneficiary_device35%ip_country_mismatch25%card_added_under_1h15%
4 · Findings Sample · synthetic data
What the report would flag.
Sample findings 3
The two systems disagree on a defined set of events.
The events in the two highest bands that the current system did not send to review are listed with their reasons in the raw scores.
Some high scores may share a device across accounts.
The data in a shadow test is not enough to say. A pilot can test it.
5 · What a pilot would measure Sample · synthetic data
The same number, live, for 30 days.
A pilot runs the engine in parallel on the flow and measures it against the number named above. It ends in a go or no-go on evidence.
The agreed number.
Share of payouts sent to manual review, reported the way you already report it.
The disagreement list.
What happened to the events the two systems read differently.
The open hypothesis.
The shared-device question from finding F-03.
Want this on your own traffic?
A shadow test is 14 days and free. You keep the report.