Allstate, Advisor Pro, and the High Stakes of Interpreting Ambiguous Data

This case is used internally at Kellogg and is not available for public distribution.

As the senior vice president of strategic distribution at Allstate, a major legacy insurance company, Don Sands was leading an initiative to develop and integrate an AI-powered conversation guide, called Advisor Pro, into its agents' point of sales systems. Advisor Pro used AI capabilities to power a "dynamic scripting" system that guided agents as they fielded inbound phone calls from potential customers. The goal was to streamline and simplify the process while helping agents understand the customer's needs more quickly. The ultimate goal was to help agents give advice that was based on---and tailored to---the specific needs of customers. After testing a prototype version of Advisor Pro in two states, Arizona and Virginia, Sands's team began testing and tweaking a more refined version of the tool in a larger cohort of states, with nine states serving as the test group. The results would be measured against a control group of 13 states using the old system.

By the fall of 2019, the first batch of data from the test had begun to come in, showing mixed results. Fully implementing Advisor Pro nationwide would be expensive and---given the results to date---somewhat risky. Sands faced a decision: Should he continue tweaking and rolling out the tool nationwide---or abandon it?