Lift chart and cumulative gains chart

Goal

Both charts are built on the ordering that a per-unit score induces, a predicted probability or any rating at all, and they answer a single question: how many responses does contacting only the best-ranked capture. The decision they serve is a budget decision: a follow-up campaign or a retention exercise has a bounded envelope and has to choose whom it treats.

The deliverable is the cutoff, expressed in deciles or as a percentage of the ranked population, with the assumptions behind it: the cost of a contact, the value of a response, the validation set the figures come from and the base rate, the share of responders in the whole population. The gains table belongs to the deliverable as much as the chart does: a cutoff without its table cannot be verified.

The confusion matrix, which data mining covers, measures a model at a decision threshold that is already set: it counts true and false positives. The two charts are the instrument by which that threshold is chosen.

Usage

When to use it

• A score or a ranking is available: the model returns a propensity per unit, calibrated or not, whose ordering alone matters.
• Budget or capacity is capped: the number of cases treated is bounded and they have to be chosen, whether the setting is campaign targeting, a retention exercise or a work queue for debt collection, fraud control or file review.
• Contact cost and response value can be quantified: the profit curve becomes computable and the cutoff follows from it.
• Two candidate models are compared: on the same population, the chart that dominates captures more responses for the same budget.
• A trade-off goes to a committee: the chart reads without any knowledge of the model and is discussed in CHF.

When not to use it

• Binary output with no underlying score: a rules engine supplies no ranking, so read the confusion matrix at the decision threshold the rule already imposes.
• A continuous target: a forecast of volume or amount does not convert into responses captured, so use a forecast error measure.

Description

The ranking, raw material for both charts

A supervised classification model returns a propensity, the probability of response it predicts. Turning that into a binary decision requires a decision threshold, and Provost and Fawcett observe that many business decisions do without this step: it is enough for them to order the population and act from the top down until the budget runs out (Data Science for Business). Cutting the ranked list at a rank amounts to setting that decision threshold indirectly; the cutoff dealt with here is that rank, expressed as a share of the population. The calibration of the score is therefore immaterial: a score that overstates every probability draws the same charts as an accurate one, as long as it orders the population the same way.

The cumulative gains chart

The population sorted by decreasing score is cut into equal-sized bands: ten deciles by direct-marketing convention, percentiles on large populations, where the plot smooths out. The horizontal axis carries the cumulative share of the population contacted starting at the top of the ranking, in steps of 10% up to 100%. The vertical axis carries the cumulative share of the actual responses captured in that leading band. A point reads like this: at 30% of the population contacted, the vertical axis gives the share of responders reached. Jaffery and Liu (2009) document this reading and the split into deciles it comes from.

The diagonal y = x is the reference: contacting X% of the population at random captures X% of the responses on average. It is the null model, and Fawcett makes it the common foundation of evaluation by ranking, ROC analysis included (2006). The curve of a useful model is concave and sits above it; it departs furthest from the diagonal in the first deciles and rejoins the point (100%, 100%): contacting everyone captures everyone, whatever the model.

Lift, under two distinct constructions

Lift measures how much better than chance a band does. It is computed in two ways that practice regularly conflates, and each serves a different decision.

Cumulative lift is the ratio between the cumulative gains chart and the diagonal: cumulative share of responses divided by cumulative share of the population. Plotted against that same share, it starts high and decreases, because each band added is less dense in responses than the one before. At the last point it is exactly 1.00, since numerator and denominator are both 100% there.

Decile lift is computed band by band, without cumulation: response rate of the band divided by the base rate. It is plotted as bars against the decile number. It is the marginal measure, independent of the bands that precede it.

The distinction governs the choice of cutoff. Cumulative lift converges to 1.00 by construction, so it stays above 1 long after the point where extending the campaign costs more than it brings in. A reader who has only that curve sees a model still better than chance and concludes that the campaign should go on. The decision variable is decile lift or, better, the marginal profit of the band.

Building the table, decile by decile

1. Score every unit in the population with the model, then sort by decreasing score.
2. Cut the sorted list into equal-sized bands, ten by convention.
3. Count the actual responses in each band, taken from a labelled validation set.
4. Compute the response rate of the band: its responses divided by its size.
5. Divide that rate by the base rate to obtain the decile lift.
6. Add up the responses of bands 1 to k, then set that total against all responses to obtain the cumulative gain.
7. Divide the cumulative gain by the cumulative share of the population to obtain the cumulative lift.
8. Multiply the band's responses by the value of a response and subtract its size multiplied by the cost of a contact: that is the marginal profit, whose running total gives the profit curve.

The eight computations give the eight columns of the gains table. Plot the cumulative gains chart with its diagonal, then the lift chart while stating which of the two constructions is drawn: a legend that says only "lift" leaves the reader guessing which one.

The cutoff: break-even point and profit curve

Two figures close the calculation: the cost of a contact and the value of a response. They come from cost accounting and from the business, the same figures a cost-benefit analysis consumes, and cost divided by value gives the break-even point. A band whose own response rate exceeds that point adds profit; a band below it destroys profit, including where the running total stays favourable. Set against the base rate, the break-even point also reads as a lift: 6.25% on a population that responds at 9.3% is a lift of 0.67, the height of the break-even line on a decile lift chart.

The profit curve accumulates the marginal profit band by band, and its maximum designates the cutoff. As long as the response rate decreases from band to band, which is what a working ranking produces, stopping at the peak amounts to stopping at the last band whose own rate clears the break-even point: a running total stops growing exactly when the term added turns negative. The reasoning comes from the choice of cutoff under asymmetric costs and uncertain distributions set out by Provost and Fawcett (Machine Learning, 2001); the profit curve is the form of it a business audience can read (Data Science for Business). With no quantifiable contact cost or response value there is no cutoff to optimise, and the ranking is used as it stands.

What distorts the reading

A lift measured on training data is optimistic. Overfitting flatters the first decile there, and the first question to ask in front of a chart is which set it comes from.

A low base rate makes the deciles noisy. A lift of 4.0 computed on twelve responses is indistinguishable from a lift of 2.5, by sampling variance alone. The safeguard is an exploratory analysis of the raw counts per band, before any percentage is read.

Two populations with different base rates give incomparable lifts. The same model shows a higher lift on a population where the event is rare, so comparing two campaigns means setting each cumulative gains chart against its own diagonal or normalising.

The table is filled with observed responses. Multiplying a predicted probability by the size of the band and treating the result as a number of responses produces an imaginary profit, which the cutoff inherits.

Operational capacity outranks the arithmetic. The optimal cutoff and the achievable cutoff do not fall in the same place, and bringing them together, by spreading the campaign out or by revising its scope, is business analyst work.

The cutoff is recomputed for every campaign. It depends on the contact cost and the response value, which drift, as well as on the base rate, which moves from one exercise to the next. Carrying over the previous cutoff unchanged amounts to deciding on stale assumptions, with nothing in the chart to signal it.

AI considerations

The clearest gain is producing the table. The eight columns are repetitive coding work whose specification fits on one page, and a language model writes a first pass in SQL or Python that the analyst reads back against the definitions. The second use is translation: explaining to a committee why the campaign stops at the fifth decile, starting from the table and the two assumptions, is the kind of rephrasing a language model does well.

Machine learning platforms produce these charts by default, which moves the work to two acceptance questions: which set the chart was computed on and which of the two lift constructions is drawn.

A language model given the cumulative gains chart alone comes back with a confident cutoff, read off the elbow of the curve, an argument from shape where the decision is economic: neither the cost of a contact nor the value of a response appears in the chart. Whether the validation set resembles the population that will be contacted stays a judgement about the organisation and its data.

Examples

A Swiss health insurer is preparing a retention campaign on its complementary insurance portfolio, ahead of the cancellation deadlines. A model scores 20'000 policyholders on their propensity to accept a retention interview and the population is cut into ten deciles of 2'000. A contact costs CHF 25, adviser call and handling included. A policyholder who stays is worth CHF 400, the acquisition cost avoided plus the net margin on one year of premium. The base rate observed on the validation set is 9.3%, 1'860 responders.

The break-even point is 25 ÷ 400, that is 6.25%. Decile 5 clears it with 7.0%, decile 6 fails with 5.0% and cumulative profit peaks at CHF 398'000 then decreases band by band. Cumulative lift is still 1.54 at the sixth decile and falls to 1.00 only at the tenth: at no point does it signal where to stop. The cutoff retained is therefore five deciles, 10'000 policyholders out of 20'000.

The retention desk places 800 calls a week and eight weeks remain before the deadline, so 6'400 contacts against the 10'000 the cutoff calls for. Capacity decides ahead of the arithmetic: the campaign stops at the third decile, 6'000 policyholders, for CHF 362'000, CHF 36'000 below the optimum. The business analyst prices the two routes that recover those 36'000 and leaves the sponsor to arbitrate: bringing the launch forward by five weeks takes capacity to 10'400 contacts; raising the pace to 1'250 calls a week covers the 10'000 in eight weeks.

What the business analyst hands over fits on one page: the cutoff, five deciles or three under the capacity constraint; the gains table; the assumptions of CHF 25 and CHF 400 with their provenance; the validation set and the base rate; the date of the next recomputation.

Visualisations

The cumulative gains chart with its diagonal shows what the ranking earns, and the marker placed on the cutoff attaches the decision to it. The two lift constructions, side by side on the same data, show the cumulative curve above 1 while the marginal bars have already dropped below the break-even point. The profit curve states the cutoff in francs, the unit a committee decides in. The gains table carries the figures the chart does not show.

Cost

Tooling

A spreadsheet is enough as soon as the scores are exported: ten rows, eight columns and two native charts. The computation stays visible in front of a committee. On large populations, the split into bands is done in SQL with the NTILE window function. In Python, scikit-learn supplies the scores and matplotlib the plot; scikit-plot outputs both charts in one line. In R, the ROCR and lift packages cover the same need. SAS Enterprise Miner and IBM SPSS Modeler draw them without any code. The profit curve is almost always drawn by hand.

Sources

• Jaffery, T. and Liu, S. X., Measuring Campaign Performance by Using Cumulative Gain and Lift Chart, SAS Global Forum 2009, Paper 196-2009: the standard construction of the artefact, axes, split into deciles, the lift formula and campaigns handled with the two charts side by side. Both charts belong to direct-marketing practice, which has used them since the 1990s, and this paper documents their received form.
• Fawcett, T., An Introduction to ROC Analysis, Pattern Recognition Letters, 27(8), 2006, pp. 861-874: the logic of ranking and cutoff common to all these evaluations, together with the diagonal as the reference for random selection.
• Provost, F. and Fawcett, T., Robust Classification for Imprecise Environments, Machine Learning, 42(3), 2001, pp. 203-231: the foundation for choosing a cutoff by expected value, when misclassification costs and the class distribution are uncertain.
• Provost, F. and Fawcett, T., Data Science for Business, O'Reilly, 2013, ch. 8: the profit curve as an artefact for a business decision and the framing of the model as a ranking tool. A synthesis work for both contributions.