Measuring False Declines You Can’t See: What a Fraud-Vendor Switch Reveals

Your fraud screen only learns it was wrong about payments it lets through, when some come back as fraud. Payments it blocks never come back at all, so you never learn how many were good customers. Switching fraud vendors is one of the few times you can find out. One switch showed what happened across 30,612 high-risk payments.

The animation above is simulated: blocked payments vanish for good, but during a vendor switch the old vendor’s would-be declines go through and reveal how they turn out.

Key takeaways

• You never learn what a blocked payment would have done. It never goes through, so it can never become a dispute. No fraud system can measure its own false declines from its own data.

• A vendor switch is a natural experiment. The old vendor usually keeps scoring every payment. Payments it would have blocked but the new vendor approves now go through, so you finally see how they turn out. It’s free, as long as you record both vendors' decisions. Split traffic randomly between the two and you can check both vendors at once.

• At one Stripe merchant, 61.5% of the payments Radar would have declined were good customers. But 38.5% were fraud, which is high enough that blocking them still made money at most margins.

• How you count changes the answer. Count every attempt Radar didn’t block, instead of only the payments that actually went through, and the same group looks 96% good.

• Fraud shows up late. Half of fraud disputes arrive within 16 days, but the last 5% take 80 to 119 days. Check too early and fraud that hasn’t arrived yet looks like good customers.

What is a false decline?

A false decline is a legitimate payment your fraud screen turned away. Real customer, real card, and you said no. (We’ve written about what each one costs. This post is about measuring them.)

The trouble starts with the phrase "false decline rate." Three different numbers go by that name:

How the numbers show up in practice

• Industry reports use share of sales. Datos Insights puts the global average at 1.51% of ecommerce sales (source).

• Merchant surveys use share of orders. In the Merchant Risk Council’s 2026 report, two-thirds of merchants put their own rate between 2% and 10% of orders (source).

• Vendor marketing uses relative cuts. "X% fewer false declines" rarely comes with the starting rate, the definition, or how good customers were identified.

A vendor quoting "2%" and another quoting "60%" can be describing the same system. Ask which one they mean. Then ask how they knew which declines were good, because that’s the hard part.

Why you can’t see your false declines

You learn a payment was fraud when the cardholder disputes it. But a cardholder can only dispute a payment that went through. A blocked payment never goes through, so you never learn whether it was fraud or a good customer.

Machine learning researchers call this the selective labels problem. Credit risk teams have wrestled with it for decades as reject inference: learn from the applicants you approved, then infer what the rejected ones would have done.

But your screen didn’t block at random. It blocked on signals your analysis may not have: a device fingerprint, a card’s history across a network, a consortium blocklist. Two payments that look identical to you can look very different to the screen, and the screen blocked the worse one. So if you estimate blocked payments from approved ones that look similar, you’ll make them look safer than they were, every time.

Backtests have the same blind spot. Replay a new rule on past payments, and the ones your current screen already blocked all show "no dispute." A simple backtest counts them as good customers the new rule would wrongly stop. You don’t know that. They just never had the chance to be disputed.

Even Stripe needs data it can’t get from blocked payments. From Radar’s documentation: “For a small subset of payments, Stripe modifies the reported risk score so we can measure the performance of our models and obtain data for subsequent model development.”

Stripe changes the score on a small share of payments on purpose, to check how its own models do. It keeps those results to itself, so they can’t tell you your false decline rate.

How to measure false declines: five methods compared

Each method trades cost against accuracy. Only the last three show real results for payments you’d normally block, and only natural experiments are free.

No single method is enough. But natural experiments are the only free way to see real results for payments you’d normally block, and most merchants throw theirs away.

The natural experiment hiding in every vendor switch

Most merchants change fraud vendors at some point, and most treat it as a migration project. But it’s also the one time two separate engines judge the same payments.

On Stripe, Radar scores every card payment unless you opt out, whether or not its rules make the final call. So when a new vendor takes over the decision, Radar keeps writing a risk score on every charge. Payments Radar would have blocked but the new vendor approves now go through, and you see how they turn out, like any other payment. The technical name is counterfactual (or off-policy) evaluation: using real results to judge decisions you didn’t make.

You don’t have to wait for the full switch. Have both engines score every payment and randomly split who decides, say 70% old and 30% new. In the 30% group, you see how payments the old engine would have declined turn out. In the 70% group, you see the same for the new engine. You get both false decline rates at once, from the same weeks of traffic.

Other natural experiments are probably already in your data:

• Overrides. Your team approved payments the vendor wanted to decline.

• Threshold changes. Raise a block rule from 75 to 85, and payments scoring 75–85 start going through, so you see how they turn out.

• Allow lists and review queues. Every payment your team approves by hand shows you how a would-be decline turns out.

• Fail-open outages. If your screen timed out and you approved by default, those payments are close to a random sample.

Radar lets "elevated" payments (scores 65–74) through by default, so you can always see how payments just below its default block line turn out. To see above the line, you need a natural experiment.

What a switch showed: 30,612 payments Radar wanted to stop

Merchant A sells digital goods on Stripe. Its Radar rule blocked payments scoring 70 or higher. That’s not Radar’s default of 75, which is why you measure a cutoff instead of assuming it. For 11 months, from October 2024 to August 2025, a second fraud vendor made the approval call after authorization. Radar kept scoring every charge.

Two things changed in the window. Radar’s block rate on 70+ attempts fell from nearly 100% to between 25% and 79% a month. And the bars got taller. Merchant A’s total attempts stayed in the same 17–31K-a-month range, but the share scoring 70+ went from under 5% to as much as 20%. Once high-risk payments could get through, more of them showed up. Keep that in mind. We’ll come back to it.

Only 68 in 1,000 ever went through

For most of those 30,612 attempts, we never found out how they’d turn out. 47.9% were still blocked by Radar, 23.9% were declined by the card issuer, and 21.4% were authorized and then canceled by the other vendor. Only 2,075, or 6.8%, actually went through and were captured. Those are the only ones that could ever show whether they were fraud.

Of those 2,075, 798 (38.5%, 95% CI 36.4–40.6%) carried a fraud signal: a fraud dispute, an early fraud warning, or a refund marked fraudulent. The other 1,277 (61.5%) were good customers.

So were Radar’s would-be declines false? Mostly, yes. Six in ten were good customers. But 38.5% fraud is 164× the rate for payments Radar scored 0–9, and 34× Merchant A’s overall rate in the same months. Two things are true at once: most of these declines would have been false, and blocking them was still the right call.

Be careful what you count as "approved"

A payment the card issuer declined can’t become a dispute. Neither can one the merchant authorized and then canceled. Count them as "approved, no dispute" and you’ve called 13,889 payments good customers when they never had the chance to be anything else.

Same payments, same months. Depending on what you count, Radar’s would-be declines look anywhere from 62% to 96% good. Before you trust a false decline rate, ask which payments it counts and what it counts as fraud.

Authorized and captured are this far apart because Merchant A’s other vendor decided after the bank had already authorized. If your payments are captured automatically, the two counts are about the same, and you can skip that step.

Fraud rate by Radar score, including scores Radar normally blocks

The blue bars are normally empty because Radar blocks those payments. The switch filled them in. One more lesson hides in the last bar: payments scoring 90–99 show less fraud (29%) than those scoring 80–89 (48%). That’s because the other vendor canceled more of the very worst payments, so the ones it let through were cleaner. You only see what the other vendor chose to let through, not every payment at that score.

Merchant B: the opposite answer

Merchant B, on a different processor, runs an enterprise fraud vendor. When the vendor said "decline unless 3DS passes," some card orders went straight to the bank without 3DS anyway. In June 2026 that was 6,337 orders. 1.75% of them carried a fraud signal (95% CI 1.43–2.07%), against 0.56% on card orders the vendor approved that month. That’s 3.1× the fraud rate of approved orders. But 98.2% of them were still good customers.

Riskier than average isn’t the same as worth blocking

So should Merchant A have kept blocking that band, when six in ten were good customers? A block is a bet. If the payment is fraud, blocking saves the loss. If it’s good, blocking costs the margin, and often the customer. Blocking a group of payments makes money when its fraud rate p is above a line set by your own margins and fraud costs:

m = margin on a good order · L = cost of an approved fraud · both as multiples of order value

At a 30% margin, with fraud costing 1.3× the order, the line sits at 18.8%. Merchant A’s 70+ band (38.5% fraud) clears it easily. Merchant B’s orders that skipped 3DS (1.75%) don’t come close. Same question, opposite answers, and you only find out which merchant you are by measuring. Turning this line into thresholds gets its own post.

Six ways a natural experiment lies to you

1. The other engine picks what you see. It approves the payments it thinks are safest, so the fraud rate you measure is probably lower than the true rate at that score. Treat it as a floor, not an estimate. Merchant A’s 90–99 dip is this effect.

2. Fraudsters notice. Let more risky payments through and more fraudsters show up. At Merchant A, the share of attempts scoring 70+ went from under 5% to as much as 20%. Check for months with attacks before you draw conclusions.

3. Fraud shows up late. Fraud disputes keep arriving for months. Check too early and fraud that hasn’t arrived yet looks like good customers.

4. Authorization isn’t approval. Only payments that were actually captured can be disputed. Count issuer declines and canceled authorizations as "approved" and you’ll invent false declines.

5. Some fraud never becomes a dispute. Count early fraud warnings and fraud refunds, not just disputes. On Merchant A’s let-through payments, that moves the fraud rate from 30.1% to 38.5%. And when a payment passed 3DS, the card issuer usually takes the fraud loss, so it may never reach you as a dispute.

6. Two engines, two populations. Compare engines on the same payments. Measure each engine’s real cutoff (Merchant A’s was 70, not 75), and check that the old engine really scored every payment the whole time before you trust a comparison. A random split fixes most of this, because both groups get the same weeks of traffic.

How long Merchant A’s fraud disputes took to arrive (point 3):

From one natural experiment to every blocked payment

A natural experiment only shows you one slice: payments one engine wanted to block and the other let through. For everything else, like payments both engines blocked, next quarter’s traffic, or a rule you haven’t shipped, you still have to estimate. The slice you can see is best used to check and correct those estimates:

1. Hide. Take the payments you can see and hide how they turned out.

2. Estimate. Run your usual estimate on them, as if they’d been blocked.

3. Compare. Actual fraud ÷ estimated fraud is your correction factor. Multiply your estimates for the payments you can’t see by it.

• One correction per merchant, per engine, per window. It measures what the blocking engine knew that your model didn’t. That differs for each engine and shifts during attacks, so measure it again each time.

• Plan for fraud cases, not payments. How precise your number is depends on how many fraud cases you have: about 100 for ±20%, about 400 for ±10% (95% confidence). Merchant A’s 798 put its 38.5% within about ±2 points.

• Say how much to trust it. Report at least $X when you couldn’t calibrate and $X ± N% when you could.

How to run your vendor switch as an experiment

If you’re about to switch fraud vendors, the experiment costs almost nothing. Skipping it costs the only free look you’ll get.

• Split, don’t just switch. Have both engines score every payment and randomly assign who decides, say 70% old and 30% new. Each group shows how the other engine’s would-be declines turn out, so you measure both false decline rates at once instead of waiting for the full switch. On Stripe, tag each payment’s group in metadata and apply Radar’s block rules only to its group.

• Split by card or customer, not by payment. Hash a stable key, so a fraudster declined in one group can’t retry into the other.

• Size the smaller group by fraud cases. The 30% group collects results on the old engine’s would-be declines more slowly. Check how long it takes to reach your minimum number of fraud cases before you start.

• Can’t split? Shadow first, and keep the old engine scoring. Run the new engine in score-only mode for 8–12 weeks while the old one decides. When you switch, turn off the old engine’s block rules, not its scoring.

• Log both decisions on every payment, keyed by PaymentIntent ID, with both scores, both thresholds, and the group.

• Let a small random slice through, with a fixed fraud budget. It’s the only way to see how payments both engines would block turn out. Assign it by the same stable key so nobody can game it.

• Write the rules down first: captured payments only; fraud means a dispute, an early fraud warning, or a fraud refund; measure 120 days after the payment; set a minimum number of fraud cases.

• Don’t loosen rules while fraud is rising. Let through a small slice in one score range, or send those payments to 3DS instead.

Five questions to ask any fraud vendor about its false decline numbers

1. Which definition: share of declines, share of good customers, or share of sales?

2. Measured on which payments: captured, authorized, or attempted?

3. How did you decide which blocked payments were good, if at all?

4. How long after the payment did you check for fraud?

5. What’s the confidence interval, and how many fraud cases is it based on?

Find your own natural experiment in Stripe Sigma

You might already have one. If your Radar rules were ever relaxed, if you approved payments from the review queue, or if another vendor made decisions for a while, some high-scoring payments got through. This query shows their fraud rate by score band.

-- Fraud rate for payments Radar scored high but you captured anyway.
-- Stripe Sigma (Trino SQL). Risk scores need Radar for Fraud Teams.
with graded as (
  select c.id,
         floor(c.outcome_risk_score / 10) * 10 as score_band
  from charges c
  where c.captured
    and c.outcome_risk_score is not null
    -- old enough for fraud disputes to arrive
    and c.created < current_timestamp - interval '120' day
    and c.created >= current_timestamp - interval '485' day
),
fraud as (
  select charge_id from disputes where reason = 'fraudulent'
  union
  select charge_id from refunds where reason = 'fraudulent'
)
select g.score_band,
       count(*) as captured_payments,
       count(f.charge_id) as fraud_cases,
       round(100.0 * count(f.charge_id) / count(*), 2) as fraud_rate_pct
from graded g
left join fraud f on f.charge_id = g.id
group by 1
order by 1

• Bands at or above your block threshold will be empty unless something let payments through. Empty means you have no natural experiment, and you’d need a holdout.

• Add early fraud warnings to catch fraud that never became a dispute.

• No Radar for Fraud Teams? Group by outcome_risk_level instead.

FAQ

What is a good false decline rate?

It depends on the definition and on your margins. As a share of ecommerce sales, industry estimates average about 1.5%. As a share of your declines, a high number can still be right: 61.5% of Merchant A’s Radar 70+ would-be declines were good customers, and blocking them still paid.

How do you calculate false decline rate?

Divide false declines by everything your screen declined (the share of declines that were good) or by legitimate attempts (the share of good customers declined). The hard part is knowing which blocked payments were good. You need real results from a natural experiment or a random holdout, or estimates you’ve checked against one.

How long should I wait before measuring false declines?

At least 90 to 120 days after the payment. At Merchant A, half of fraud disputes arrived within 16 days, 90% within 49, and the last one after 119.

Can I measure false declines with Stripe Radar alone?

Only where Radar’s blocks didn’t apply: the elevated band it allows by default, payments approved from review, allow-listed payments, or periods when your rules were relaxed. Stripe doesn’t share results from its own score tests with merchants.

How do I compare two fraud vendors fairly?

On the same payments, over the same window, with the same rules for what counts as fraud and how long you wait. The cleanest way is a random split: both score every payment, and a random slice of customers gets each one’s decision. If you can’t split, run the new vendor in shadow first and keep the old one scoring after the switch. Either way, log both decisions on every payment.

How big does a fraud holdout need to be?

Size it by fraud cases, not payments. About 100 fraud cases give a ±20% read at 95% confidence. About 400 give ±10%. At a 10% fraud rate, 100 cases means about 1,000 let-through payments, and a fraud budget to match.

See what your blocked payments are hiding

Corgi Labs shows how your blocked payments would have turned out, using your own Stripe data: shadow mode, real results for would-be declines, and estimates checked against them for each merchant. Get a free diagnostic

Related reading

• Your Fraud System Is Your Most Expensive Revenue Leak: what a false decline really costs

• Stripe Radar vs Custom ML: why network models miss your best customers

• What Corgi Labs Actually Does on Top of Stripe Radar: shadow mode and A/B design in practice

• How to Calculate Your VAMP Ratio: the fraud side of the threshold math

References

1. Stripe. Transaction risk prevention (Radar risk scores, thresholds, and score modification).

2. Lakkaraju, Kleinberg, Leskovec, Ludwig, Mullainathan. "The Selective Labels Problem: Evaluating Algorithmic Predictions in the Presence of Unobservables." KDD 2017.

3. Tocker. "Virtual Control Group: Measuring Hidden Performance Metrics." arXiv:2208.12941, 2022.

4. "Fighting Sampling Bias: A Framework for Training and Evaluating Credit Scoring Models." arXiv:2407.13009.

5. Signifyd. How the top retailers measure fraud false declines (citing Datos Insights and the Merchant Risk Council).

6. Sardine. How to measure false positives in fraud systems that hide them.