Use cases · Fintech & neobanks
Subscription tier upgrade prompt timing.
The team ships the winner. A 14% slice is being prompted into cancel.
Free-to-paid upgrade prompts are the single highest-leverage growth surface in fintech subscriptions. The standard A/B test reports the average uplift across the free-tier base, and quietly absorbs the cohort the prompt is pushing into downgrade or cancellation. On every audit we've run, that cohort exists. It is also where most of the recoverable revenue lives.
Worked audit
Lyra · Series-B neobank · ~$18M ARR · MENA + EU · 482,711 free-tier accounts
LYRA-2026Q1-AUDIT-002
Projected impact
+$1.4M / yr ARR
1 · What the team reported
Upgrade prompt timing test: week-4 (control) vs week-2 (variant). Upgrade rate 4.1% → 4.4%, p = 0.02. Team called it "winner, ship week-2" and rolled to 100% of free-tier users.
The post-test write-up read positive across the board. Internal note flagged a small bump in cancellations, attributed to noise.
2 · What our re-analysis found
Doubly-robust re-evaluation surfaced a clear asymmetry. The positive-uplift cohorts (balance-high × spend-regular, balance-high × spend-irregular) account for ~58% of the base and showed lifts of +11.7% and +5.1% respectively, larger than the global +6.4%.
But: a 14% slice of the base (balance-low × spend-irregular) showed a −8.2% lift (CI [−13.5, −2.9]). That cohort is being prompted into downgrade or cancellation. The global t-test averaged the loss against the gains and reported the winner.
3 · Why the t-test missed it
The aggregate upgrade-rate metric is dominated by the larger, healthier cohorts. The negative-uplift cell is small enough that its loss doesn't move the global mean detectably, but its dollar cost is asymmetric because cancellations destroy future LTV, not just this month's upgrade.
Per-segment CATE makes the loss visible on the same data, before it compounds. ESS guardrail keeps us from over-claiming on the overlap-limited new-account cohort (<30d tenure), which we flag for re-test rather than re-analyse.
4 · What we'd recommend
Personalise the prompt. Send to positive-uplift cohorts (~58% of base), suppress for the balance-low × spend-irregular cell. Lifts net upgrade rate from +6.4% to an estimated +11.7% and preserves retention on the cancellation-prone cohort.
Estimated annualised lift: +$1.4M ARR · retention preserved on the irregular-spend cohort.
Doubly-robust readout · week-2 vs week-4 · bootstrap 1,000 reps
| Cohort | DR estimate | 95% CI | ESS | Verdict |
|---|---|---|---|---|
| All free-tier accounts | +6.4% rel. | [+3.0, +9.8] | 0.58 | positive, larger than t-test |
| Balance-high × spend-regular | +11.7% rel. | [+7.2, +16.3] | 0.51 | clear positive uplift |
| Balance-high × spend-irregular | +5.1% rel. | [+1.0, +9.3] | 0.48 | small positive |
| Balance-low × spend-regular | +2.4% rel. | [−2.1, +6.9] | 0.42 | inconclusive |
| Balance-low × spend-irregular | −8.2% rel. | [−13.5, −2.9] | 0.36 | clear negative, prompted into churn |
| New accounts (<30d) | −3.1% rel. | [−7.2, +1.0] | 0.31 | overlap-limited; flag for re-test |
Read the full audit, then audit your own test.
Same shape we'll send back on your last A/B test — free, in three business days.