Research Details

Abstract

We present a novel approach to causal measurement for advertising, namely to use exogenous variation in advertising exposure (RCTs) for a subset of ad campaigns to build a model that can predict the causal effect of ad campaigns that were run without RCTs. This approach -- Predictive Incrementality by Experimentation (PIE) -- frames the task of estimating the causal effect of an ad campaign as a prediction problem, with the unit of observation being an RCT itself. In contrast, traditional causal inference approaches with observational data seek to adjust covariate imbalance at the user level. A key insight is to use post-campaign features, such as last-click conversion counts, that do not require an RCT, as features in our predictive model. We find that our PIE model recovers RCT-derived incremental conversions per dollar (ICPD) much better than the program evaluation approaches analyzed in Gordon et al. (forthcoming). The prediction errors from the best PIE model are 48%, 42%, and 62% of the RCT-based average ICPD for upper-, mid-, and lower-funnel conversion outcomes, respectively. In contrast, across the same data, the average prediction error of stratified propensity score matching exceeds 491%, and that of double/debiased machine learning exceeds 2,904%. Using a decision-making framework inspired by industry, we show that PIE leads to different decisions compared to RCTs for only 6% of upper-funnel, 7% of mid-funnel, and 13% of lower-funnel outcomes. We conclude that PIE could enable advertising platforms to scale causal ad measurement by extrapolating from a limited number of RCTs to a large set of non-experimental ad campaigns.

Type

Working Paper

Author(s)

Brett Gordon, Robert Moakler, Florian Zettelmeyer

Date Published

2024

Citations

Gordon, Brett, Robert Moakler, and Florian Zettelmeyer. 2024. Predictive Incrementality by Experimentation (PIE) for Ad Measurement.