The problem it solves

Members spend ¥66,000/yr. Non-members spend ¥24,000/yr. The naive read is "the program adds ¥42,000 per member." That's almost certainly wrong because of selection bias — people who join loyalty programs are systematically different from those who don't. They might be higher-income, more brand-loyal, or more frequent shoppers regardless of any program. If those factors alone explain the gap, the program adds nothing; it just attracts customers who were going to spend a lot anyway.

Matched cohort exists to strip out selection bias.

The core idea

For every member, find a non-member who looks just like them on everything observable — except program membership. Compare what the member did to what their matched counterpart did. If members systematically out-perform their matches, that difference is more plausibly attributable to the program.

The technique comes from epidemiology, where you can't randomize patients into "gets the drug" vs "doesn't" if the drug is already available. So you match treated patients to similar untreated ones on age, sex, severity, etc., and compare outcomes. Same logic here.

How matching actually happens

Pick a set of matching variables — things observable about a customer before they joined. Typical for retail loyalty:

• Pre-enrollment spend (¥ in 6–12 months before join date)
• Pre-enrollment frequency (visits per month)
• Pre-enrollment basket size
• Pre-enrollment category mix
• Tenure as customer
• Demographics (age band, postcode, gender if available)
• Channel mix (store vs online)
• Recency (when they last transacted)

Then run a matching algorithm. Two common approaches:

• Nearest-neighbor matching — for each member, find the non-member with smallest distance across all variables.
• Propensity score matching — compute each customer's "probability of being a member" given their characteristics, then match members to non-members with similar scores.

Both produce member–non-member pairs that look statistically indistinguishable on the matched variables. Example pair:

These two look identical before the join date. The only meaningful difference: Maki joined the program in early 2024, Yuki did not. Repeat for every member.

Estimating lift from the matched set

Compare post-enrollment outcomes:

Notice the counterfactual jumped from ¥24,000 (whole non-member average) to ¥51,500 (matched non-member average). That ¥27,500 difference is the selection bias — the part of the original gap that wasn't program-caused. Matched cohort surgically removes it. The remaining ¥14,500 is what we attribute to the program.

What it controls for, and what it doesn't

Controls for: anything matched on. Differences on those dimensions are netted out by construction.

Doesn't control for: anything unobservable or not matched. The residual bias sources to worry about:

• Unobservable preferences — a customer about to fall in love with the brand may join because they're about to spend more, not the reverse.
• Concurrent life events — the member moved closer to the store, got a raise, had a baby. The matched non-member didn't.
• Joining-moment commitment — at the moment of joining, the member is in "engaged mode." That energy alone might drive short-term lift independent of program mechanics.

This residual bias is why matched cohort tends to over-estimate true causal lift. The matchable variables are a partial proxy for the unmatchable ones. A program that genuinely drives +20% might show +28% in matched cohort and +175% naive.

Why ±5pp confidence interval, not narrower

Three sources of statistical noise:

• Sample size — even with hundreds of thousands of pairs, sampling variance on the mean difference is ±1–2pp.
• Matching imperfection — matches are close but not identical; residual within-pair difference adds noise.
• Period sensitivity — a single 12-month window catches macro effects (seasonality, competitor moves). Different windows give different estimates.

The ±5pp confidence interval bundles these. Honest interpretation: "the true matched-cohort lift is between 23–33% — but the true causal lift could be different still, because methodological bias isn't captured in this CI."

Why it's still the standard

• Uses data you already have — no experimental design needed.
• Peer-reviewed methodology, defensible.
• A step-change better than naive comparison.
• Can be re-run quarterly without disrupting the program.

To do better, you need experiments — and most programs don't run them, because the operational cost of denying customers loyalty is real. The next reference unpacks that.

Operating posture

When the dashboard says "+28% lift, 95% CI 23–33%":

• For planning (budgets, targets): use the point estimate, 28%.
• For risk-aware decisions (will the program still be worth it under stress?): use the lower bound, 23%, which gives net P&L of ~¥1.27B instead of ¥1.6B.
• For pushing back on the +175% naive number: the matched-cohort estimate is the defense — "yes, members spend 175% more, but most of that is who they are, not what we did."
• For improving the estimate: argue for a small holdout (see next reference).

What it is

Randomly select a subset of customers who would qualify for the program (would see the sign-up prompt, fit the eligibility criteria, etc.) and exclude them from it entirely. Compare what these excluded customers do over time to comparable customers who did get the program. The difference is unambiguously program-attributable, because the only thing that differed between the two groups was random assignment.

Why it's the gold standard

Randomization eliminates all selection bias — observable AND unobservable. With matched cohort, you control for what you can measure. With a holdout, you control for everything by construction, because the two groups are statistically identical except for one variable: program access.

Same reason randomized controlled trials are gold standard in medicine. Observational studies can suggest causality; only randomization can prove it.

How it actually works

• Pick the moment of exposure — every customer who hits the sign-up landing page, or every new transaction without a member ID, or every cart-completion event.
• Apply randomization at that moment — e.g. 1% of those interactions are silently routed to a "no program" experience.
• Track both groups over a fixed period (12 months is typical). Treatment group: enrolled in program. Control: never offered it.
• At the end, compare any outcome metric: spend, frequency, retention, basket size.

The difference between treatment and control on that metric is the causal lift.

Why most programs don't run them

Three real costs:

• Lost revenue / margin — customers in the control group might have spent more if they'd joined. You're deliberately leaving money on the table to learn.
• Customer experience — some excluded customers will be confused or frustrated ("I tried to join but couldn't"). Requires careful UX to avoid creating real annoyance.
• Political resistance — someone has to defend "we deliberately denied these customers the program" to executives, partners, and possibly regulators.

These costs are real. The dashboard flags "not running" rather than pretending the gap doesn't exist.

The hybrid approach (recommended)

You don't have to do a giant experiment to get value. Small holdouts work:

• Hold out 1–2% of new opt-ins for 12 months (a few thousand customers depending on volume).
• Track them as a control group.
• Compare to a matched sample of joiners.
• Once you have a holdout-based lift estimate, calibrate your matched-cohort estimate against it.

Example: matched cohort says +28%. Holdout says +21%. Calibration factor = 21/28 = 0.75. Future matched-cohort estimates can be multiplied by 0.75 to approximate the true causal estimate, without running a fresh experiment every time.

This pattern is becoming standard practice at retailers with mature data science teams. The small holdout doesn't have to be permanent — a single 12-month window gives you a defensible calibration factor.

Practical implementation considerations

• Power calculation — how big does the holdout need to be? Depends on baseline spend variance and expected effect size. Typical retail loyalty: a few thousand customers per arm to detect lifts of 5pp or more.
• Randomization point — must happen before the customer makes a join decision. Randomizing among existing members is too late.
• Non-compliance — some control-group customers will find other ways to join (different channel, different store). Need to handle "intent to treat" vs "treatment received" properly.
• Reporting — at the end of 12 months, the lift estimate has a confidence interval too — typically narrower than matched cohort's, but still non-zero.

The honest summary

Matched cohort is the best estimate from observational data. Holdout is the best estimate from experimental data. The deepest version of "what's the program worth?" is the holdout number — which is why "not running" is the single most consequential line on the methodology note above.