Running experiments with a real control group

The problem of measuring without a counterfactual

A retention campaign is launched targeting the highest-risk customers, and at close it's reported that 70% stayed. That sounds good, but without a comparable group that didn't receive the campaign, there's no way to know how many of those customers would have stayed anyway. Many "successful" programs are measuring regression to the mean.

Minimum design of a useful experiment

It doesn't take complex statistical machinery, it takes discipline on five points.

• A hypothesis written before launch, with the metric and the minimum relevant effect
• Random assignment of the eligible population between treatment and control
• A sample size sufficient to detect that minimum effect
• A measurement window defined in advance, not cut short when convenient
• Analysis of the incremental effect, not the absolute performance of the treated group

Scale only what demonstrated incrementality

Discipline pays off when applied to the scaling decision. A program that shows three points of incrementality with a known cost can be defended before the committee with numbers. One that only shows absolute performance is defended with a narrative, and gets cancelled at the first budget cut.

Institutionalize the practice

Organizations that learn faster don't run more sophisticated experiments: they run more experiments, document the ones that fail, and maintain a repository of what was learned. The asset isn't each test, it's the accumulated body of knowledge.