Cohort Retention and CLV: Method

Retention Matrices, Kaplan–Meier, the Shifted-Beta-Geometric Model and Customer Lifetime Value

Cohorts and the retention matrix

A cohort is the group of customers acquired in the same period (a month, a quarter, a year). Following each cohort forward gives the number of its original customers still active one, two, three periods after acquisition; dividing by the cohort size gives the retention curve of that cohort, and stacking the cohorts gives the retention matrix. Because the newest cohort has been observed for one period and the oldest for many, the matrix is a triangle. The average curve the tool reports is weighted by cohort size over the cohorts observed at each period: Σ active_ik / Σ size_i over the cohorts i that have reached period k. It is the right summary when the cohorts share one churn process; when they do not (a change of channel, price or product), the per-cohort fits show it.

Three inputs are accepted. A retention table in wide form (cohort, size, active_1, active_2, …, ragged) or long form (cohort, period, active, the period-0 row carrying the size) is read as counts of a contractual or subscription business, where a departure is observed. Transaction rows (customer, date, amount) are turned into monthly cohorts by the month of each customer's first purchase; here a customer is "active" in a month with at least one purchase, which is an activity measure, not survival (a customer may skip a month and return). For the survival analysis and the churn model the tool therefore uses the last-purchase lifetime: a customer whose last purchase lies before the last observed month is taken to have left in the following month, and one who bought in the last month is still active (censored). Customers who return after a long gap are counted as lost by that rule; the activity matrix shows them coming back. A single retention curve (period, % retained) is accepted for the model fit alone.

Churn, retention and Kaplan–Meier survival

The retention rate for period t is the share of the customers active at the end of period t − 1 who are still active at the end of t; the churn rate is one minus that. Pooling the cohorts, the customers at risk in period k are those active in period k − 1 of every cohort observed in period k, n_k, and d_k of them leave. The Kaplan–Meier product-limit estimate of the survivor function, S(k) = Π_{j ≤ k} (1 − d_j/n_j), is the probability that a customer survives k periods, using every cohort for as long as it has been observed: a cohort that has not reached period k is censored, not thrown away, which is what makes the estimate use the whole triangle rather than the oldest cohort only. Greenwood's formula gives the variance, var S(k) = S(k)² Σ_{j ≤ k} d_j/(n_j(n_j − d_j)), and the tool shows S ± 1.96 se clipped to 0–100 %. With complete data (one cohort followed to the end) the Kaplan–Meier curve is the empirical retention curve itself.

Why the aggregate retention rate rises

Retention curves of almost every customer base show a retention rate that rises with tenure. Fader and Hardie (2007) show that no change in individual loyalty is needed to produce it. Suppose each customer renews with a constant probability 1 − θ of her own, and θ varies across customers. The high-θ customers leave early, the survivors are increasingly the low-θ ones, and the aggregate retention rate climbs by selection alone (Vaupel and Yashin's "ruse of heterogeneity"). Fitting a flexible curve to the observed points ignores this mechanism and projects badly: Berry and Linoff's linear, quadratic and exponential fits to the High End data of Table 1 miss the year-12 survival by 81 %, 92 % and 30 %, while the probability model below misses it by 4 %.

The shifted-beta-geometric (sBG) model

Lifetime T (the period in which the customer leaves) is shifted geometric given θ, P(T = t | θ) = θ(1 − θ)^t−1, and θ follows a beta(α, β) distribution across customers. Integrating θ out gives

P(T = t) = B(α + 1, β + t − 1)/B(α, β), S(t) = B(α, β + t)/B(α, β), r_t = S(t)/S(t − 1) = (β + t − 1)/(α + β + t − 1),

computed without beta functions by the forward recursion P(T = 1) = α/(α + β), P(T = t) = P(T = t − 1)(β + t − 2)/(α + β + t − 1). The mean churn probability across customers is E(θ) = α/(α + β). The shape of the beta distribution is the diagnostic (their Figure 3): α, β both below 1 is U-shaped (a loyal group and a group that leaves at once), α < 1 ≤ β reverse-J (most customers churn little, a sizeable sub-segment leaves early: the usual case), α ≥ 1 > β J-shaped (most customers churn a lot), both above 1 an interior mode (fairly homogeneous churn, the geometric limit as both grow large).

The parameters are estimated by maximum likelihood. For a single curve given as percentages the cohort size cancels: the dropout fractions S(t − 1) − S(t) weight ln P(T = t) and the surviving fraction weights ln S(T) (their Appendix B, which the tool reproduces: LL = −2.116 at α = β = 1 and the maximum −1.611 at α = 0.668, β = 3.806 for the High End segment). For a multi-cohort table the log-likelihood of Fader and Hardie's multi-cohort note is Σ_i Σ_t (n_i,t−1 − n_it) ln P(T = t) + Σ_i n_i,Ki ln S(K_i), the customers still active at each cohort's last observed period being censored; the tool maximises it by Nelder–Mead on (ln α, ln β) from several starts. The constant-churn alternative is the shifted geometric with θ̂ = dropouts / customer-periods at risk; its log-likelihood is reported beside the sBG's (one parameter fewer), together with the root-mean-square and mean absolute percentage errors of S(t) on the observed average curve. When the sBG runs to very large α and β the data carry no heterogeneity and the two models coincide. A cohort's active count that rises from one period to the next (a data error, or returning customers in an activity matrix) is kept in the matrix but counted as no dropout in the likelihood.

The projection to the entered horizon is S(t) from the fitted parameters. With a single curve, "fit through" holds the later points out: the error of the projection at those periods is the honest test of the model, the fit statistics on the fitted points are not. The Berry and Linoff data of Fader and Hardie's Table 1 fitted to years 1–7 project year 12 within 4 % (High End) and 2 % (Regular) of the observed value; the constant-churn model is 30–70 % low.

Lifetime, discounted lifetime and CLV

The expected lifetime counting the acquisition period is Σ_{t ≥ 0} S(t) = (α + β − 1)/(α − 1) for α > 1 and infinite for α ≤ 1 (the tail of near-zero-churn customers is heavy enough); the median lifetime is the first period with S ≤ 50 %. The present value of a new customer's margins, m per active period booked at the start of the period and discounted at d per period, is

E[CLV] = m Σ_{t ≥ 0} S(t)/(1 + d)^t = m · DEL, DEL = ₂F₁(1, β; α + β; 1/(1 + d)),

the discounted expected lifetime of Fader and Hardie (2010), finite for any d > 0. The tool sums the series directly to a relative term of 10⁻¹⁵ and checks the sum against the hypergeometric form. DEL − 1 = Σ_{t ≥ 1} S(t)/(1 + d)^t is the multiple of the margin earned after the acquisition period, the quantity the simple formula below approximates. The period discount rate from an annual rate r with m periods a year is d = (1 + r)^1/m − 1.

For a customer who has already been active for n periods (n − 1 renewals made), evaluated just before the next renewal decision, the discounted expected residual lifetime is

DERL(n) = Σ_{t ≥ n} [S(t)/S(n − 1)] (1 + d)^−(t−n) = ((β + n − 1)/(α + β + n − 1)) ₂F₁(1, β + n; α + β + n; 1/(1 + d))

(Fader and Hardie 2010, eq. 6). It rises with n because the survivors are the low-churn customers: with α = 3.8, β = 15.2 and d = 10 % it is 3.31 periods for a new customer and 3.84 for one active five periods. The residual value of the customer base is Σ over the active customers of m · DERL(n), the tool taking each cohort's customers active in its last observed period at tenure n = observed periods + 1; Fader and Hardie's example gives $120,543 for 34,178 customers at $1 a period. Ignoring the heterogeneity (valuing every customer at the average retention rate) understates this value, often by a third or more, which is the "peril" of their title.

For transaction rows the margin per active customer-period is the revenue per active customer-month (acquisition month included) times the gross margin percentage unless a margin is entered. The payback period is the first period at which the cumulative margin per original customer (the size-weighted average, then the sBG projection m · S(t) beyond the observed periods) reaches the acquisition cost, interpolated within the period; LTV/CAC is m · DEL divided by CAC, where 3 or more is the usual target for a subscription business.

The simple margin formula

Gupta, Lehmann and Stuart (2004) value a customer base with CLV = m · r/(1 + d − r), the margin multiple for a constant retention rate r and discount rate d: 1.07 to 5.00 for r between 60 % and 90 % and d between 16 % and 8 %, 4.09 at r = 90 % and d = 12 %. It is the geometric (constant-churn) case of the sBG multiple, DEL_geo − 1 = (1 + d)/(1 + d − r) − 1, and it is what an analyst uses when only an average retention rate is known. The tool prints it beside the sBG multiples so the difference, which is the value of the survivors' lower churn, is visible.

Limits

The sBG is a model of a discrete-time contractual setting: renewals at fixed intervals, departures observed. For continuous-time contracts (a utility cancelled on any day) its analogue is the exponential-gamma (Lomax) model, and for non-contractual purchase data (the transaction rows here), where a departure is never observed, the Pareto/NBD and BG/NBD models are the proper tools; the last-purchase lifetime used here is a working approximation that counts a long pause as a departure. The model assumes each customer's churn probability is constant over time; a base whose individuals become more loyal with tenure (or less) needs the beta-discrete-Weibull. Pooling cohorts assumes one churn process; the per-cohort fits are the check. The constant margin m ignores growth in spend per customer, cross-selling and cost-to-serve; the Kaplan–Meier intervals treat customers as independent and the last transaction month as complete.