Nav: Home

How measurable is online advertising?

March 21, 2019

Key Takeaway: Current "observational" approaches to digital advertising measurement may not be as effective as large-scale randomized control trials.

CATONSVILLE, MD, March 21, 2019 - Researchers from Northwestern University and Facebook in March published new research in the INFORMS journal Marketing Science that sheds light on whether common approaches for online advertising measurement are as reliable and accurate as the "gold standard" of large-scale, randomized experiments.

The study to be published in the March edition of the INFORMS journal Marketing Science is titled "A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook," and is authored by Brett Gordon of Northwestern University; Florian Zetttelmeyer of Northwestern University and the National Bureau of Economic Research; and Neha Bhargava and Dan Chapsky of Facebook.

"Our findings suggest that commonly used observational approaches that rely on data usually available to advertisers often fail to accurately measure the true effect of advertising," said Brett Gordon.

Observational approaches are those that encompass a broad class of statistical models that rely on the data "as they are," generated without explicit manipulation through a randomized experiment.

"We found a significant difference in the ad effectiveness obtained from randomized control trials and those observational methods that are frequently used by advertisers to evaluate their campaigns," added Zettelmeyer. "Generally, the current and more common methods overestimate ad effectiveness relative to what we found in our randomized tests. Though in some cases, they significantly underestimate effectiveness."

Measuring the effectiveness of advertising remains an important problem for many firms. A key question is whether an advertising campaign produced incremental outcomes: did more consumers purchase because they saw an ad, or would many of those consumers have purchased even in the absence of the ad? Obtaining an accurate measure of incremental outcomes ("conversions") helps an advertiser calculate the return on investment (ROI) of the campaign.

"Digital platforms that carry advertising, such as Facebook, have created comprehensive means to assess ad effectiveness, using granular data that link ad exposures, clicks, page visit, online purchases and even offline purchases," said Gordon. "Still, even with these data, measuring the causal effect of advertising requires the proper experimentation platform."

The study authors used data from 15 U.S. advertising experiments at Facebook comprising 500 million user-experiment observations and 1.6 billion ad impressions.

Facebook's "conversion lift" experimentation platform provides advertisers with the ability to run randomized controlled experiments to measure the causal effect of an ad campaign on consumer outcomes.

These experiments randomly allocate users to a control group, who are never exposed to the ad, and to a test group, who are eligible to see the ad. Comparing outcomes between the groups provides the causal effect of the ad because randomization ensures the two groups are, on average, equivalent except for advertising exposures in the test group. The experimental results from each ad campaign served as a baseline with which to evaluate common observational methods.

Observational methods compare outcomes between users who were exposed to the ad to users who were unexposed. These two groups of users tend to differ systematically in many ways, such as age and gender. These differences in characteristics may be observable because the advertiser (or its advertising platform) often has access data on these characteristics and others, e.g., in addition to knowing the gender and age of an online user, it is possible to observe the type of device being used, the location of the user, how long it's been since the user last visited, etc. However, the tricky part is that the exposed and unexposed groups may also differ in ways that are very difficult to measure, such as the users underlying affinity for the brand. To say that the ad "caused" an effect requires the research to be able to account for both observed and unobserved differences between the two groups. Observational methods use data on the characteristics of the users that are observed in attempt to adjust for both the observable and unobservable differences.

"We set out to determine whether, as commonly believed, current observational methods using comprehensive individual-level data are 'good enough' for ad measurement," said Zettelmeyer. "What we found was that even fairly comprehensive data prove inadequate to yield reliable estimates of advertising effects."

"In principle, we believe that using large-scale randomized controlled trials to evaluate advertising effectiveness should be the preferred method for advertisers whenever possible."
-end-
About INFORMS and Marketing Science

Marketing Science is a premier peer-reviewed scholarly marketing journal focused on research using quantitative approaches to study all aspects of the interface between consumers and firms. It is published by INFORMS, the leading international association for operations research and analytics professionals. More information is available at http://www.informs.org or @informs.

Institute for Operations Research and the Management Sciences

Related Facebook Articles:

Can seeing the Facebook logo make you crave social media?
A new study examined how social media cues such as the Facebook logo may affect frequent and less frequent social media users differently, sparking spontaneous hedonic reactions that make it difficult to resist social media cravings.
Facebook likes don't make you feel better
Receiving 'likes' on social media posts doesn't make people feel better about themselves or improve their mood if they are down.
Many new doctors may be posting unprofessional content on Facebook
When researchers searched Facebook for the public accounts of all urologists who graduated from US residency programs in 2015, they found that a substantial proportion of these accounts contained self-authored unprofessional content based on the professionalism guidelines of three physicians' organizations.
Why we underestimate time when we're having fun on Facebook
Updating your Facebook status can be a fun way to while away the hours -- but now it seems it really is making us lose track of time as we do it.
One in 5 adults secretly access their friends' Facebook accounts
Most people are concerned about the prospect of their social media accounts being hacked, but a new University of British Columbia study finds that it's actually people we know who frequently access our accounts without our permission.
Northeastern and Facebook partner on next generation research projects
Face­book announced on Wednesday that North­eastern is one of 17 research uni­ver­si­ties selected to partner with the com­pany on next gen­er­a­tion, joint tech­nology projects.
Feeling blue? Taking a break from Facebook might help
A new study shows that regular use of social networking such as Facebook can negatively affect your emotional well-being and satisfaction with life.
Live long and... Facebook?
Is social media good for you, or bad? Well, it's complicated.
What are the health risks of having a different 'Facebook self'?
People may express their true self more easily on Facebook than in person, and the more one's 'Facebook self' differs from their true self, the greater their stress level and the less socially connected they tend to be, according to a new study published in Cyberpsychology, Behavior, and Social Networking.
How effective are personalized political ads on Facebook?
In this election year, political advertising on social networking sites such as Facebook is increasingly making use of personalized ads to target specific social media users.

Related Facebook Reading:

Best Science Podcasts 2019

We have hand picked the best science podcasts for 2019. Sit back and enjoy new science podcasts updated daily from your favorite science news services and scientists.
Now Playing: TED Radio Hour

Climate Crisis
There's no greater threat to humanity than climate change. What can we do to stop the worst consequences? This hour, TED speakers explore how we can save our planet and whether we can do it in time. Guests include climate activist Greta Thunberg, chemical engineer Jennifer Wilcox, research scientist Sean Davis, food innovator Bruce Friedrich, and psychologist Per Espen Stoknes.
Now Playing: Science for the People

#527 Honey I CRISPR'd the Kids
This week we're coming to you from Awesome Con in Washington, D.C. There, host Bethany Brookshire led a panel of three amazing guests to talk about the promise and perils of CRISPR, and what happens now that CRISPR babies have (maybe?) been born. Featuring science writer Tina Saey, molecular biologist Anne Simon, and bioethicist Alan Regenberg. A Nobel Prize winner argues banning CRISPR babies won’t work Geneticists push for a 5-year global ban on gene-edited babies A CRISPR spin-off causes unintended typos in DNA News of the first gene-edited babies ignited a firestorm The researcher who created CRISPR twins defends...