Research Details

Abstract

Randomized controlled trials (RCTs) have become increasingly popular in both marketing practice and academia. However, RCTs are not always available as a solution for advertising measurement. In practice, RCTs can be technically difficult or even impossible to implement in many settings. Since the need to measure advertising effects does not go away when RCTs are infeasible, what should advertisers do? Using a representative sample of 1,009 experiments at Facebook, we attempt to answer this question using two non-experimental approaches. First, we use stratified propensity score matching, a method drawn from the program evaluation literature. This method performs poorly, with an average prediction error exceeding 1,500% of the causal effect of advertising. This poor performance comes despite using an extensive set of user-level covariates, a sophisticated machine learning model to estimate the propensity score, and over one million users per RCT. Our second approach uses predictive models based on simple proxy metrics, with the unit of analysis at the level of an RCT. This approach performs much better, achieving prediction errors between 30% and 60%. Finally, we examine whether an advertiser would reach the same conclusion about a campaign's success using either an RCT or a non-experimental method. We find that our best proxy metric-based predictive approach would lead an advertiser to reach the same conclusion in 80% to 95% of cases.

Type

Working Paper

Author(s)

Brett Gordon, Robert Moakler, Florian Zettelmeyer

Date Published

2021

Citations

Gordon, Brett, Robert Moakler, and Florian Zettelmeyer. 2021. Close Enough? A Large-Scale Exploration of Non-experimental Approaches to Advertising Measurement.