When fraud history knows the future: the two clocks inside a fraud feature
How future dispute data leaks into fraud features, why lookback windows don’t fix it, and how to rebuild history around outcome event times.
Bo HuangEngineering, Corgi Labs
8 min read
A payment happens on Monday. Another arrives on Wednesday using the same card. The first payment is disputed three weeks later. When we rebuild Wednesday’s history, does that dispute count?
It can, even in a rolling window that excludes Wednesday’s payment. Monday is in the past. The dispute is not.
At one digital-goods merchant, we looked at 3,810 payments whose card had an earlier purchase, within 180 days, that was eventually disputed. For 96.7% of them, the correct count at scoring time was zero: none of those disputes had been filed yet. The retrospective feature gave them a positive count anyway.
Counted another way, as contributions (one earlier disputed purchase in one later payment’s history), only 281 of 18,402 had been filed in time. About 98.5% of that history was still in the future. Short bursts weigh heavily in this count: 8,203 contributions came from purchases less than 24 hours apart.
In the illustration, a payment scored on day 5 already shows one prior dispute under the payment clock, though none had been filed. The outcome clock counts it only for scores after the dispute is filed on day 12.
Key takeaways
The rule is simple: an outcome may enter a historical feature only if it was filed before the payment being scored was created. The earlier payment’s own timestamp is not enough.
No lookback window enforces that rule. In our card audit, 24-hour and 90-day windows alike counted mostly disputes that had not yet been filed.
At one merchant, 96.7% of payments with disputed card history should have counted zero at scoring time. The retrospective feature counted more.
Retrospective history inflated a backtest’s mean ROC-AUC from 0.64 to 0.79. Retraining on event-time history averaged 0.71, though the variation between test months is too large to call that gain significant.
The rule applies to every grouping, including broad cohorts such as card brand. A check before publishing keeps it from regressing.
The two clocks inside a fraud feature
The target label answers what eventually happened to a payment. Historical features answer what was available when deciding about a later one. A delayed dispute can supply a valid training label while arriving too late to supply a historical feature.
A feature such as "previous disputes on this card" is usually built by filtering to earlier purchases and attaching their dispute flags. The filter proves only that the purchases were earlier. Their present-day dispute flags can still carry outcomes filed after scoring time.
| Clock | Question it answers | Suitable use |
|---|---|---|
| Payment creation | Had this purchase already happened? | Prior payment volume and spend |
| Outcome event | Had this dispute, warning or refund occurred? | Prior outcome history |
This is temporal leakage in a feature: information unavailable at prediction time is used to build the model (Kaufman et al., 2012).
The rule follows directly: an outcome may enter a feature only if its own event time is earlier than the creation time of the payment being scored. Below, filed means the outcome’s event time and scoring time means the creation time of the payment being scored. A lookback window sets how far back to look, not what was known. Strictly, what matters is when we learned of the outcome. Production adds a few hours of ingestion delay on top of the event time (see Scope and limits).
The rule cuts both ways. Imagine several purchases on one card within an hour, all disputed three weeks later. A retrospective rebuild gives each repeat the earlier purchases' eventual disputes, so a count that should read zero climbs to several. Now imagine the same card used again months after a disputed purchase: that dispute was filed before the new payment, so it is legitimate history.
In the figure, the burst’s later purchases count six earlier disputes and know none of them at scoring time. The return visit counts one dispute, filed 89 days earlier.
Count what was known at scoring time
To measure exposure directly, we paired each scoring payment with every earlier payment on the same card that was eventually disputed, then asked whether that dispute had already been filed. We repeated the count for lookback windows from 24 hours to 180 days.
Changing the window doesn’t fix the leak. At 24 hours, none of 8,203 contributions was known at scoring time. At 90 days, 195 of 18,090 were. A longer window adds older history, but most of what it adds is still in the future.
Most of the leak comes from short repeats. Among 5,319 seven-day pairs where both payments were eventually disputed, every earlier dispute was filed after the later payment was scored. In 97.2% of those pairs, the two filings were within 24 hours of each other, consistent with related purchases being disputed in a batch. Because one dispute can contribute to several later payments, these shares describe the history the model learns from, not all traffic.
Watch the missing information arrive
We then kept the same 180-day histories and moved only the observation cutoff forward. At scoring time, 1.5% of the retrospective count was visible. After 30 days, 62.6% was visible. After 60 days, 85.5%. After 120 days, 99.9%.
This curve measures the problem; it isn’t the fix. A score can’t wait 120 days, and no fixed delay makes payment-time history safe. A feature that counted only purchases older than 90 days would still pick up disputes filed after scoring time, and it would ignore recent disputes already on record. Only the event-time comparison is exact.
Label maturity is a separate question; this curve follows history, not labels. For labels, the experiment below kept only payments at least 120 days old, so their own disputes had time to be filed.
Rebuild the history around outcome events
Our first repair excluded the current payment from its own lifetime windows. That stopped a payment from counting its own outcome, but an earlier purchase could still carry a dispute filed after scoring time.
The full repair places payments and outcome events on one timeline per entity and applies the rule directly: at each payment, the feature counts only outcome events filed strictly before scoring time. The lookback window then applies to the events, so "disputes in the last 28 days" means disputes filed in those 28 days. This is the point-in-time principle used in historical feature retrieval (Feast).
Each outcome needs its own clock. Disputes and refunds use their creation times. An early fraud warning can exist without a dispute, so it uses its own timestamp. A final lost status becomes known at closure; attaching it to the filing date backdates the result. Families whose timing couldn’t be supported were excluded from model inputs.
The rule has no exemptions by grouping. We had banned unsafe outcome families for individual entities but exempted broad cohorts such as card brand and funding type, on the theory that they track no single customer. Those cohorts still inherited other customers' future outcomes. In one recorded run, admitted columns showed 26.96× and 11.11× univariate lift in training, yet the serving table read zero for every card brand: its windows cover recent payments, and none of those had been charged back yet. Trained on future outcomes, served nothing. Closing the exemption removed 40 columns.
A fix like this can quietly regress, so a check runs before rebuilt features replace the published ones. It recomputes lifetime dispute, early-fraud, and refund counts by payment time and by outcome time, then compares the candidate only on rows where the two differ. Rows where they agree, such as a card with no disputes at all, would let a leaky feature pass. On a 10% sample, the candidate must match the event-time count on 99.9% of those rows, or the build stops.
Train on the history the scorer will receive
We compared three versions of the same model across 19 monthly test periods, each trained on six earlier months.
A backtest built on retrospective history reported mean ROC-AUC of 0.79. Tested on the history the scorer would actually receive, the same model scored 0.64: it had learned to rely on history that isn’t there at scoring time. Retrained on event-time history, it averaged 0.71. A chronological split doesn’t fix this: a later test month still contains future information if its features were assembled retrospectively.
Scope and limits
The census and availability curve come from saved audit outputs for one merchant’s card histories. A matched email or customer census was not run.
The AUC results come from an offline notebook, not the production evaluation pipeline. They are fold means with standard deviations of about 0.11–0.12. Paired per-fold differences were not saved, so the gain from 0.64 to 0.71 is not a tested difference.
Training labels in that experiment use each payment’s final dispute status, including disputes filed after the test month began. This affects all three arms alike, so the comparison stands, but all three absolute values are optimistic.
The reconstruction uses each outcome’s source timestamp. Production scoring reads history from a periodically refreshed snapshot, so it learns of an outcome later than that. In the same audit, a snapshot six hours stale undercounted the event-time value by 4.6%. That’s small next to the leak, but even event-time training stays slightly optimistic, most of all for short windows.
References
Kaufman, Rosset, Perlich and Stitelman (2012). Leakage in data mining: Formulation, detection, and avoidance. ACM Transactions on Knowledge Discovery from Data, 6(4), Article 15.
Feast. Point-in-time joins.
scikit-learn. Common pitfalls: data leakage.