Is OTT the Star of the Show?
Incorporating OTT into your user acquisition and growth strategy, along with adopting the appropriate analytics mindset, can enhance your LTV:CAC (Lifetime Value to Customer Acquisition Cost) ratio.
Alwin Haensel, PhD, Marketing Measurement & Analytics
Jean-Michel Boujon, Growth consultant, advisor and mentor
Content
1. Summary
2. Introduction
3. What data is available to measure OTT success?
4. First steps easy wins with OTT?
5. What is the true Customer Acquisition Cost (CAC) for OTT?
6. Conclusion
Summary
After running several OTT campaigns over the last six years, we wanted to share with you what we have learned and how you can best approach the channel to grow your business efficiently.
Our analysis delves into various aspects of OTT advertising, including available data for measuring success, its position in the consumer journey, and its influence on other channels.
Regarding customer acquisition cost (CAC) measurement, we explore three different models: baseline 7-day view on IP matching, Multi-touch Attribution (MTA) on IP matching, and Incrementality Testing/Conversion Lift Experiments. Each model offers unique strengths and weaknesses, providing varying perspectives on OTT’s effectiveness.
While OTT advertising presents promising opportunities for marketers, navigating its complexities requires a multifaceted approach to measurement and attribution. By leveraging a combination of data-driven models and experimentation, marketers can gain deeper insights into OTT’s impact and optimize their strategies accordingly.
Definition
Over-the-Top advertising (OTT) advertising refers to the placement of advertisements on streaming video content delivered over the internet, bypassing traditional television distribution methods like cable or satellite. OTT advertising allows marketers to reach audiences who consume content through OTT platforms such as streaming services (Hulu), smart TVs (Pluto TV), gaming consoles, and mobile devices. These ads can take various forms, including pre-roll, mid-roll, or post-roll ads.
There are two forms of OTT:
1- Connected TV (CTV): Content is primarily viewed on television screens, either through smart TVs or devices connected to TVs. Users typically have a more passive viewing experience similar to traditional TV watching, often sitting back and enjoying longer-form content on a larger screen.
2- Online Video (OLV): Content can be viewed on a wider range of devices, including desktops, laptops, tablets, and smartphones. Users may have the ability to engage with the content through likes, comments, shares but the ad unit is mostly non-clickable.
Introduction
The landscape of video streaming is evolving at a remarkable pace, with an impressive 73% year-over-year growth. As forward-thinking advertisers continuously seek out new and scalable acquisition channels, Over-The-Top (OTT) emerges as an enticing option to explore.
What distinguishes OTT is its capacity to deliver your message to an engaged audience within a non-skippable environment. This unique feature offers a prime opportunity to capture the attention of viewers and drive meaningful engagement with your brand.
However, it’s essential to acknowledge that OTT is still in its infancy as an industry. Consequently, there is a notable absence of standardized attribution metrics that marketers can rely on to accurately assess the true value of this channel.
For performance marketers like us, navigating this uncharted territory requires a concerted effort to establish robust measurement frameworks capable of quantifying the impact of OTT within our marketing mix. While we’re accustomed to leveraging platform data for giants like Google and Facebook, the question arises: should we adopt a similar approach for OTT?
In our pursuit of answers, this white paper aims to provide insights into how we can tackle attribution challenges specific to OTT and offer practical tips to optimize our advertising budgets effectively.
It’s worth noting that the insights shared within this document are drawn from our firsthand experiences managing multiple OTT campaigns across various platforms for diverse clients. While our experiences may not encapsulate the entirety of the industry, we hope they serve as a catalyst for critical reflection on OTT attribution methodologies.
Finally, let us remember that attribution is not a quest for perfection but rather a journey toward uncovering truths. There is no silver bullet in attribution; instead, there are tools and methodologies that can bring us closer to understanding the impact of our marketing efforts. Embrace the journey, but don’t let the pursuit of perfection consume your soul.
What data is available to measure OTT success?
Through our OTT media provider, we procure comprehensive logs encompassing each unique OTT ad impression, delineating essential details such as timestamp, hashed IP address, device operating system, user-agent specifics, network data, ad creative particulars, impression cost, and zip code. An illustrative example is provided below.
We did not distinguish residential and non-residential IPs in the following evaluations. We can assume that approx. 70% of the ad delivery is going to a residential audience. By not distinguishing between residential and non-residential IPs, we likely tend to overstate the OTT performance a bit as non-residential IPs can be accessed by several people, sometimes hundreds and increase the likelihood that the IP will convert at a higher rate.
Position of OTT in the consumer journey
The overarching objective of the campaign was focused on prospecting rather than retargeting. We conducted a compelling analysis encompassing all users, leveraging IP address matching to link site visits with OTT ad impressions.
Below, we present the analysis findings pertaining to various levels of user site engagement and the corresponding timing of OTT ad impressions. Here’s an illustrative example of a customer journey involving OTT interactions:
As we see in the above example OTT ad impressions tend to happen in the beginning of the customer journey. Overall we could match 3% of the IP addresses with OTT ad back to site visitors and 0.7% could be matched back to converting users. We further analyzed all customer journeys where we could match OTT ad impressions and site visits, and wanted to see when the OTT impressions are happening relative to the site funnelstep event.
First, we looked how often the user saw an OTT ad days before the respective site event. As shown in the below table, in 95–98% of the cases the OTT impressions happen days before the site event, as desired. We do see that the average number of impressions is relatively high to get a site visit (22.4 ad impressions on average), but only 8.4 ad impressions total to convert to a sale.
From this data, it is very clear that audience targeting and frequency capping becomes a key factor of success. As a marketer, I am only interested in the 8.4 impressions targeting the right audience to get a sale. I am not interested in the 22.4 impressions targeting the wrong audience to only get a site visit.
Next, we checked how often OTT ads are shown to site visitors days after they have been to the site. Here, we observed surprisingly high numbers. 30–40% of site visitors are exposed to OTT ads days after they have visited the site. Note here that we didn’t analyze the impact of running retargeting ads in the path to conversion but we understand this would be a valuable exercise.
For Site visit:
- 95% of users see ads days before the site visits. (22.4 impressions on average)
- 21% of users see ads on the day of the site visit (5.3 impressions on average)
- 37% of users see ads days after the site visits (35.2 impressions on average)
Unexpected retargeting can be encountered, during incomplete or delayed installation of the vendor pixel, leading to an effective expansion of the retargeting pool. If the vendor fails to differentiate individuals who have visited the site through the pixel, inadvertent retargeting may occur within your campaign
Interesting are the following stats on the number of OTT ads seen by converting users:
- On average a converting user sees just 8.4 OTT impressions
- 83% of converting users with OTT ads in their customer journey see at most 10 OTT before the conversion, on average they see just 3 OTT ads (!)
- 87% of converting user see at most 15 OTT ad impressions before the conversion
- 91% of the converting user see at most 20 OTT ads before making the conversion
- Only 7% see more than 30 ads, but they see on average 64 OTT ad impressions
Influence of OTT on other channels (Paid and Non-Paid)
Over the years, I’ve frequently heard discussions about how OTT primarily affects direct traffic. Upon observation, we’ve noticed comparable channel mix profiles whether customer journeys are exposed to OTT ads or not. Consequently, we can infer that OTT influences paid channels just as significantly as it does non-paid channels such as direct and SEO.
Upon analysis using a Multi-touch Attribution model, we’ve found that exposure to OTT ads doesn’t significantly alter the breakdown of channels attributed to conversions. Regardless of OTT exposure, the share of conversions from Organic and Branded campaigns remains stable at approximately 52%. This suggests that OTT isn’t favoring one channel over another in terms of attribution.
Therefore, if a vendor is assigning attribution credit from Direct or branded SEO to OTT, it’s overlooking the holistic view. It’s essential to recognize that direct traffic ownership is often ambiguously defined within marketing teams, making it susceptible to misattribution without accountability.
In essence, while OTT plays a role in the marketing mix, it’s crucial to ensure that attribution practices consider the entire spectrum of channels to accurately reflect contributions to conversions.
First steps easy wins with OTT?
Frequency cap
The primary goal of your OTT vendor is to maximize impression sales while maintaining favorable performance metrics. It falls upon you to ascertain the optimal number of ads to deliver to each user in order to achieve your desired outcome. By comprehending user behavior and implementing a frequency cap, you can effectively optimize your campaign.
Retargeting traffic
Discuss with your OTT vendor how to cut the fat to reduce impressions on users who are unlikely to convert. We see that most of the OTT affected users (95–98%) see the ads before they reach the specific site engagement level, but there is also a quite high amount of 30–36% where the users see also OTT ads after they have visited the site. It’s essential to deploy your vendor pixel early to enable them to exclude previous site visitors from your prospecting audience.
What is the true Customer Acquisition Cost (CAC) for OTT?
We are looking at three different measurement models in this section. You will see that CAC can vary a lot depending on which approach you are using, all have different strengths, levels of sophistication but also weaknesses.
Baseline: 7-day view on IP matching
It attributes the entire conversion to OTT, as soon as we measure a touch-point link based on IP address between the OTT ad and the conversion event within the attribution window of 7 days.
Limitations
- IP — user matching limitation, we only link OTT impressions to sales via observed IP links. Technical user recognition limitations arise from shared IPs, VPNs or proxies and dynamic IPs, etc.
- This method tends to provide a huge overvaluation of the impact of a simple OTT ad impression on the buying decision of the customer, neglecting any other marketing interaction in the customer journey.
Benefits
- This measurement model is by default and for free provided by most OTT providers
- It provides an out-of-the box measurement for the conversion influence rate of OTT ads.
Results
This approach enables us to provide a fast first CAC measurement of the OTT channel. In the following sections we will use the 7-day view evaluation as a benchmark to compare to other more sophisticated methods.
Multi-touch Attribution (MTA) on IP matching
Multi-touch attribution (MTA) is a method used to measure marketing effectiveness by considering all touchpoints or interactions in a customer’s journey. MTA assigns partial credits to each touchpoint, totaling to 100%, revealing the impact of each channel or campaign in a complex multi-channel environment.
Limitations
- IP — user matching limitation, we only link OTT impressions to sales via observed IP links. Technical user recognition limitations arise from shared IPs, VPNs or proxies and dynamic IPs, etc.
- MTA cannot measure the impact of offline channels and has limitations when it comes to measuring impression heavy channels.
- In terms of the OTT, we have to make assumptions on the engagement value of the OTT ad impression compared to site-visit sessions.
- MTA is generally more likely to measure correlations rather than casualties
- Complexity: Multi-touch attribution models can be complex to implement and interpret, especially for organizations with diverse marketing channels and customer touchpoints.
Benefits
- MTA provides continuous and dynamic measurements
- Multiple MTA providers are available
- MTA is strong in capturing the impacts of different parts in the CJ
- You have immediate results after setup
Results
Using the IP address matching logic for OTT ad impressions with site visits, we can incorporate OTT into the multi-touch attribution model. The logic was as follows, for every converting user, we checked based on the IP address if we have OTT ad impressions in the data and if so, we add this touch-point to the customer journey and this can then receive attribution results from the multi-touch attribution model.
Example Customer Journey MTA evaluation:
Our MTA logic is based on a three-phase customer interaction concept and it considers the timing of the individual touch-points within the customer journey, the timing of a touch-point to the conversion event and the engagement depth within the conversion funnel. OTT ad impressions are treated similar to bouncer sessions with just one pageview, in order to reflect that just seeing an ad, can not bring the customer deeper into the funnel, it merely helps to create awareness and hence traffic.
The resulting CACs are comparable with Paid Search Non-Brand. Considering in addition that we only covered with this OTT->IP->MTA a fraction of the users, where we could build the IP linking — the results seem too good to be true.
There is a huge difference in the resulting customer acquisition costs (CAC) between the measured MTA results and reported performance from the OTT provider.
Incrementality Testing / Conversion Lift Experiments
Incrementality testing, or uplift testing, is a key method in performance marketing to accurately measure the impact of campaigns. By randomly assigning individuals to either a treatment (exposed to the campaign) or control group (not exposed), marketers can determine the additional value or ‘lift’ generated by their efforts. This method relies on causal inference to establish a cause-and-effect relationship and often employs synthetic control, a technique that constructs a control group using historical data. Ultimately, incrementality testing provides marketers with actionable insights to optimize their campaigns.
Limitation
- Good lift experiments are time, resource and cost intensive to setup and to execute.
- External factors beyond the experiment setup can influence the results a lot.
- The results give a measurement at a given point in time, and not dynamic performance evaluation.
- Lift experiments do not give exact evaluations, but rather confidence intervals of possible values and some mean results.
Benefits
- A well setup and executed experiment can measure the true impact of the marketing “treatment”
- Lift experiments are able to measure the uncertainty of treatment effects
Results
We actually wanted to know not just the general value of OTT, but what the incremental effect of the separate two main OTT sub-areas CTV (ads on big screen smart TVs) and OLV (online video ads) actually is.
The experiment was setup as a three-cell with hold out:
- Holdout 33% of market
- Cell A with only CTV ads, 33% of market
- Cell B only OLV ads, 33% of market
Experiment duration was 14 days, US nationwide. Cell geo split was done via DMA areas.
Lift experiment results:
In cell A (CTV ads) we measured a strong Lift with strong statistical significance, cell B (OLV ads) showed only a small Lift with only a low significance.
Below, we show the resulting CAC estimates for both OTT versions, by MTA, the OTT provider and the results from the Lift experiments.
Note that lift experiments give us estimates with a confidence range, reflected below with the error bars.
Note, OTT vendors often also offer the ability to perform lift experiments within the platform. This is a nice way to obtain fast and in-expensive estimation results. But for a true evaluation, an independent and tailored experiment setup and evaluation is usually highly recommended.
Finally, we want to compare the OTT evaluation based on the lift experiment to performance results of other platforms. We see that CLV ads are better performing than Meta or Youtube.
Conclusion
We favor the Lift testing methodology for evaluating the effectiveness of OTT advertising. While Mix-media modeling is also a viable approach for estimating OTT’s value within your media mix, we didn’t employ this method during our test. However, with Lift testing, we observed that Connected TV (CTV) outperformed Online Video (OLV), YouTube, and Meta platforms. It’s important to note that these results may vary based on factors like budget allocation and creative execution and industry. .
Although OTT may not surpass Paid Search in terms of performance, it serves as a valuable channel for expanding brand reach, especially when direct response channels like Paid Search, Affiliate, and SEO have reached their saturation points. While incorporating OTT may initially increase your blended Customer Acquisition Cost (CAC), the subsequent boost in brand awareness can lead to the development of more direct traffic over time.
If you’re already running ads on Meta platforms but haven’t explored OTT, considering testing CTV could be beneficial, potentially competing with your Meta CAC. In conclusion, it’s essential to test the OTT channel and utilize lift experiments to gauge its effectiveness accurately.
Contact Jean-Michel Boujon for a consultation if you need to grow your business efficiently.
Contact Alwin Haensel for questions related to your customer data infrastructure, 1st party event tracking, multi-touch attribution, or general analytics questions regarding marketing measurement.
