Take Action

Home | Faculty & Research Overview | Research

Research Details

Close Enough? A Large-Scale Exploration of Non-experimental Approaches to Advertising Measurement

Abstract

Randomized controlled trials (RCTs) have become increasingly popular in both marketing practice and academia. However, RCTs are not always available as a solution for advertising measurement. In practice, RCTs can be technically difficult or even impossible to implement in many settings. Since the need to measure advertising effects does not go away when RCTs are infeasible, what should advertisers do? Using a representative sample of 1,009 experiments at Facebook, we attempt to answer this question using two non-experimental approaches. First, we use stratified propensity score matching, a method drawn from the program evaluation literature. This method performs poorly, with an average prediction error exceeding 1,500% of the causal effect of advertising. This poor performance comes despite using an extensive set of user-level covariates, a sophisticated machine learning model to estimate the propensity score, and over one million users per RCT. Our second approach uses predictive models based on simple proxy metrics, with the unit of analysis at the level of an RCT. This approach performs much better, achieving prediction errors between 30% and 60%. Finally, we examine whether an advertiser would reach the same conclusion about a campaign's success using either an RCT or a non-experimental method. We find that our best proxy metric-based predictive approach would lead an advertiser to reach the same conclusion in 80% to 95% of cases.

Type

Working Paper

Author(s)

Brett Gordon, Robert Moakler, Florian Zettelmeyer

Date Published

2021

Citations

Gordon, Brett, Robert Moakler, and Florian Zettelmeyer. 2021. Close Enough? A Large-Scale Exploration of Non-experimental Approaches to Advertising Measurement.

READ
KELLOGG INSIGHT

Explore leading research and ideas

Find articles, podcast episodes, and videos that spark ideas in lifelong learners, and inspire those looking to advance in their careers.
learn more

COURSE CATALOG

Review Courses & Schedules

Access information about specific courses and their schedules by viewing the interactive course scheduler tool.
LEARN MORE

DEGREE PROGRAMS

Discover the path to your goals

Whether you choose our Full-Time, Part-Time or Executive MBA program, you’ll enjoy the same unparalleled education, exceptional faculty and distinctive culture.
learn more

Take Action