From Meridian To Robyn, Your Ultimate Comparison Guide For Open-sourced MMM

Over the past few years, we've seen an increasing interest in marketing mix modeling (MMM). This is partly due to data privacy and pressure from third-party cookies, which have resulted in a resurgence in MMM.

One of the hot topics in MMM is the open-source MMM. Both Google and Meta have released their open-source Media Mix Models, which, for brevity, we also refer to as MMM.

Google's first MMM, LightweightMMM, is now replaced by Meridian. Meta's MMM model is called Robyn and is available in R and Python.

In this post, we will review these open-source models. We will discuss the differences, pros, and cons of each model.

Open-sourced MMM

LightweightMMM was an earlier open-source Bayesian marketing mix modeling (MMM) library created by Google to assist advertisers in understanding and optimizing their marketing expenditures across various media channels.

It is important to note that LightweightMMM is not an official Google product and currently lacks support. LightweightMMM is developed in Python.

Meridian serves as the successor to LightweightMMM. This open-source MMM framework allows advertisers to construct and operate their MMM models internally. Meridian is designed to provide users with a more advanced and flexible modeling experience.

Meridian is distributed under Apache License V2.0, allowing it to be used, modified, and distributed freely. Meridian can be used for commercial purposes without any restrictions.

Robyn is another noteworthy open-source MMM package developed by Meta. It aids businesses in evaluating the incremental impact of media spending on sales and other key performance indicators (KPIs).

Robyn is crafted to be an accessible tool for organizations eager to implement Media Mix Modeling (i.e., a limited version of Marketing Mix Model that uses only media channels), while focusing on reducing bias and promoting sound decision-making.

Robyn is available under the MIT open-source license, which permits free use for personal and commercial purposes. It was initially implemented in R, but Meta released a Python package for it, which could lead to a broader adaptation.

This blog post discusses the key features and distinctions of these open-source MMM solutions.

Modeling and Methodology

In this section, we look at the statistical models underlying these open-source MMM products.

Bayesian Approach

While Robyn could be included in the definition of Bayesian statistics, in the context of MMM, Bayesian models are the ones that explicitly define prior distribution. In Bayesian statistics, a prior distribution represents one's beliefs about a parameter before observing any data.

For this reason, both Google's MMMs are Bayesian MMM. This offers two advantages:

• The Bayesian approach can incorporate business knowledge and experimental results using Prior distributions.
• The Bayesian approach allows uncertainty propagation in measurements.

These two advantages make the Bayesian approach an excellent framework for MMMs. This is especially important when the number of data points is limited, which is usually true for most MMM challenges.

In MMM projects, modelers frequently encounter limited data, which can often be quite noisy, highlighting the need for business knowledge incorporation and uncertainty quantification.

LightweightMMM and Meridian both facilitate hierarchical geo-level modeling. This method utilizes geo-level marketing data to deliver more accurate parameter estimates. Moreover, geo-level data can increase the number of data points for training models. While this may sound like a great idea, in practice, geo-level data may offer limited insight compared with pooled data.

Within the MMM community, the Bayesian priors are used for model calibration. This is especially useful when trying to incorporate experimental evidence into the models. This is a neat and scientifically valid approach. However, it's challenging to set up the right priors for models in practice. Moreover, by setting priors, we are artificially limiting the modeling space.

Robyn uses Ridge Regression, a classical Machine Learning method. The difference between Ridge Regression (RR) and simple regression is that RR provides a tunable penalty term that controls the model's complexity. This is known as regularization.

In the Bayesian framework, RR's regularization is equivalent to setting a Gaussian prior over the value of the unknown coefficients. Hence, Robyn can be seen as a Bayesian method. But in reality, Robyn consists of different methods working together for marketing measurement and optimization.

Unlike Bayesian methods, Robyn uses multi-objective optimization to incorporate experimental evidences, like GeoLift, into the model (see next section).

Robyn's Architecture

Meta's Robyn is based on an ecosystem of tools and algorithms that were originally developed by Meta. This includes:

• The Nevergrad package: A gradient-free optimization platform that is suitable for fast multi-objective optimization.
• The Prophet package for time-series forecasting and decomposition. This is helpful to capture trends and seasonality effects.

At its core, Robyn employs Ridge Regression (RR) to regularize parameter estimation, providing a systematic approach to address potential collinearity or multicollinearity issues commonly found in MMMs.

RR introduces a penalty term to ordinary least squares (OLS) estimates, effectively reducing the coefficients of correlated predictors. This penalty not only helps decrease the variance of these estimates but also enhances the model's stability, making it less vulnerable to multicollinearity.

Robyn uses the Prophet algorithm for time-series decomposition, which helps detect trends and address seasonal confounding factors. Within Prophet's algorithm, users can pass event data, such as holidays. Prophet is similar to a non-parametric model, which makes it idea for fast testing. However, this feature could make it less ideal for troubleshooting and additional customization.

A unique feature of Robyn's MMM is the idea of multi-objective optimization. This feature is designed to address several challenges in MMM and improve the models' statistical validity and practical applicability.

In the next section, we will discuss multi-objective optimization and its implications.

Multi-Objective Optimization in Robyn

Traditional MMM often focuses solely on minimizing statistical error, such as the Normalized Root Mean Squared Error (NRMSE), which measures how well the model fits the historical data.

Robyn expands on the NRMSE by incorporating additional objectives that reflect business considerations and practical constraints. This approach acknowledges that a model with the best statistical fit might not always be the most plausible or actionable for a business.

By optimizing for multiple objectives simultaneously, Robyn aims to produce models that are accurate, more reliable, and aligned with real-world business goals.

Robyn's multi-objective optimization typically involves balancing three key objectives:

• Minimization of NRMSE (Statistical Error): This is the traditional goal of minimizing the difference between the model's predictions and the historical data
• Avoidance of extreme results (Business Error): Decomp.RSSD stands for decomposition root sum of squared distances. It measures the distance between the model's effect share (how much a channel is said to drive sales) and the spend share (how much was spent on that channel). Minimizing this helps avoid extreme results or strategies that deviate from the status quo. This makes the model more plausible and easier for businesses to adopt.
• Minimization of Calibration Error (Experimental Error): This objective uses results from experiments to calibrate the model and to minimize the error between the model's predictions and the experimental data. This helps to address biases, inaccuracies in the data, and endogeneity issues that may arise from incorrect model specifications.

Robyn uses the Nevergrad package (Meta's own optimization algorithm) for fast multi-objective optimization, which enables the model to efficiently explore different parameter combinations and find solutions that balance all objectives.

Users can set weights for each of these objectives, which allows them to control the relative importance of each objective. For example, setting the weights to (1, 0, 0) would result in the standard approach, focusing solely on NRMSE. A weight of (1,1,1) means all objectives are considered equally by the optimization algorithm. By default, the weights are set equally across all three objectives (1, 1, 1)

Here are the advantages of multi-objective optimization:

• More Realistic Models: By including business plausibility as an objective, Robyn helps to avoid recommending extreme strategies that might be statistically sound but practically unworkable.
• Improved Accuracy: Using experimental data to include calibration error as an objective improves the accuracy of the model by aligning it with real-world outcomes and addressing biases.
• Better Decision Making: Multi-objective optimization leads to more reliable and actionable insights into marketing spending decisions. It can help optimize budget allocation across channels while avoiding extreme recommendations.
• Flexibility: The ability to set weights across different objectives allows users to customize the modeling process and tailor it to their specific business needs.

In summary, Robyn's multi-objective optimization is a valuable tool that goes beyond traditional statistical optimization by incorporating business considerations and calibration data to produce more reliable, actionable, and realistic media mix models.

Meridian Architecture

Meridian is a versatile modeling framework grounded in Bayesian causal inference. It effectively manages large-scale geo- and national-level modeling. We have previously discussed Meridian in this post.

Just like LightweightMMM, Meridian is based on a hierarchical Bayesian model that let users make use of geo-level marketing data. For more information about the model, refer to this paper from Google. But the model can be simply described as below:

Where kpi is typically the revenue, volume, sales per period, or any other KPI the business is interested in, such as website visits, alpha is the baseline sales that is independent of marketing, seasonality, or other factors.

Note that trend, seasonality, and other factors are arbitrary terms that the modeler must decide whether to use, depending on the project and business context. However, the model's ultimate goal is to determine the right coefficient for the media channels. This will enable Meridian to prescribe (i.e., optimize) the best spend allocation.

The following Flow chart is from LightweightMMM's GitHub, which describes the structure and flow of media mix modeling using this package. As we said, Meridian's model is very similar to LightweightMMM.

TensorFlow Probability

Meridian uses TensorFlow Probability to model the probabilistic econometric model. According to Google:

This means that, as an option, Meridian can be run on GPU hardware, which improves the speed of MMM inference and optimization. However, executing an MMM system on GPU could cost organizations additional money.

Media data transformations

There are three main concepts in modeling media: diminishing return, carryover effect, and lagged effect.

Merdian uses various data transformations to incorporate these effects into the final model.

The Diminishing Returns (DR) in media explains that as media spending increases, the incremental value received from it declines. In other words, media reaches a saturation point, and as a consequence of DR, reaching a higher sales target by solely increasing media spend becomes increasingly harder. See This post for more details about DR.

Merdian implements the DR effect using the Hill function, a concept rooted in Biochemistry. The Hill function has two tunable parameters per media channel.

Different examples of the Hill function (Link to the original image)

The Carryover Effect refers to the way the influence of media advertising builds up over time due to ongoing spending. Some models use the Attribution Window concept, which has many similarities to the Carryover Effect.

Merdian can analyze how previous media activations affect current sales. However, this will increase the number of parameters in the model, which can raise the risk of overfitting.

Meridian uses an Adstock function to measure the media carryover effect. The Adstock function in Meridian has one tunable parameter and one hyper-parameter, L, which determines how far the media effect accumulates back in time.

Model Calibration

The core idea of Bayesian model calibration is perhaps as old as the Bayesian theorem itself. Within the context of MMM, a publication from Google in 2024 describes the process of MMM calibration with the Bayesian Priors.

MMM calibration with Bayesian priors involves incorporating prior knowledge and experimental results into the model to improve its accuracy and reliability. This process is crucial for ensuring that MMMs accurately represent advertising effectiveness.

Here are the advantages of Bayesian Priors in MMM calibration:

• Incorporating Domain Knowledge: Bayesian MMMs can accommodate modelers' prior knowledge or beliefs about the effectiveness of various media channels.
• Using Incrementality Experiments: Incrementality experiments can be used as priors to enhance the model’s accuracy and ensure a closer alignment between the model’s outputs and the actual incremental value of the channels.
• Model Regularization: Calibration helps regularize the model by reducing uncertainty in model estimates. This prevents unrealistic Return on Ad Spend (ROAS) estimates for media channels. The regularization is a similar concept as in Robyn's Ridge Regression.

But also, here are a few challenges with MMM calibration using Bayesian Priors:

• Disparate Aspects of Advertising Impact: MMMs measure the long-term average effectiveness of ads, while incrementality experiments usually measure short-term effectiveness. As a result, incrementality experiments may not represent the modeling window of an MMM model.
• Potential for Bias: If confounding factors are not properly accounted for, bias can be introduced into the estimates.
• Identifiability Issues: When using multiple incrementality experiments for a single media, the parameters for each experiment can be unidentifiable, leading to challenges in parameter estimation.
• Identifying the Appropriate Prior is Challenging: A significant obstacle to using Bayesian Priors is the absence of a clear method for determining the suitable Prior for a model. This situation yields considerable flexibility and options for the modeler. Consequently, models might be misspecified despite potentially fitting the data effectively.

For more information on how Meridian uses Bayesian Priors in model calibration, please check out this page.

Reach and Frequency

Meridian can use reach and frequency data to more accurately model media impact. This is particularly useful when a person can be exposed multiple times, with varying impacts depending on the frequency.

Meridian's approach to reach and frequency data offers a more nuanced understanding of media impact compared to LightweightMMM's reliance on impressions alone. While LightweightMMM primarily used impressions as input, Meridian expanded this by incorporating reach and frequency metrics, which can provide a more precise measurement of marketing impact.

Impressions count the number of times an ad is displayed, regardless of whether it's the same person seeing it multiple times. This can lead to an oversimplified view of ad exposure! That is why using impressions alone doesn't account for the fact that the impact of an ad can vary depending on how frequently an individual sees it.

Therefore, Meridian incorporates reach and frequency data for certain media channels for more precise measurement. This helps Meridian measure the effects of ad saturation by differentiating between reaching new viewers and repeatedly exposing the same audience. As the following figure shows, this enables Meridian to optimize the media channel for reach and frequency (source of image in this paper).

However, it is worth noting that reach and frequency data can be estimated from a black-box model and may not always be consistent or perfectly represent ad quality. Adding reach and frequency leads to a more complex model that may require more data for accurate results.

Causal Inference in Meridian

Meridian is founded on causal inference principles, aiming to accurately assess the true impact of marketing expenditures on Key Performance Indicators (KPIs). This method is vital as metrics like Return on Investment (ROI), response curves, and budget optimization inherently suggest causality—specifically, how fluctuations in marketing spend influence results.

At its core, Meridian functions as a sophisticated regression model designed to interpret causal relationships within data. The implications of these relationships are drawn from the specific estimands and assumptions rather than the regression model itself.

Within this framework, Meridian identifies several treatment variables, which encompass paid media, organic media, and various non-media treatments. These treatment variables are crucial as they are the ones for which the model estimates a causal effect, allowing analysts to discern the impact of each element on overall performance.

In order to accurately isolate the effects of the treatment variables, Meridian incorporates control variables. These controls serve the vital purpose of estimating a baseline outcome.

According to Meridian's documentation, for the model to effectively account for confounding factors, these control variables must meet the "backdoor criterion" within Meridian's causal graph. This stipulation implies that they exhibit an upward-pointing V-shape interaction with both the spending and sales metrics, a design that helps to mitigate bias in the analysis.

Google Query Volume (GQV)

Google possesses some of the most precise query and keyword data. When you incorporate paid search into your MMM model, GQV can effectively serve as a reliable control variables.

Google Query Volume (GQV) frequently acts as a significant confounder in the relationship between media and sales. This is especially relevant for paid search, where the volume of queries can influence ad volume if campaign settings permit, like when there is no budget cap in place. To obtain unbiased causal estimates for any media affected by GQV, it is crucial to control for this factor. Neglecting to account for GQV may result in an inflated assessment of the causal effect of paid options.

In short, GQV as a control variable can reduce bias in measuring paid search ROI. This is an advantage over Meta's Robyn.

The Final Word

This article discussed two leading open-sourced MMM solutions: Google's Meridian (the successor of LightweightMMM) and Meta's Robyn. Each of the two solutions have their own pros and cons.

Most interesting about Robyn's MMM is the multi-objective optimization, allowing Robyn to incorporate business considerations and calibration data to produce more reliable, actionable, and realistic media mix models.

Meridian highlights the Bayesian modeling and Causal framework. Although Bayesian Priors assist in integrating experimental evidence into models, they can be challenging to establish in real-world situations. Next, Google's use of GQV is especially noteworthy, as it allows Meridian to assess the Paid Search channel be

With the release of Merdian, we are embarking on a new era of marketing measurements. Setting up an MMM is becoming easier, and many companies are quickly adapting to this change. While building MMM pipelines powered by Robyn or Meridian is easier than ever, the devil remains in details.

Don't navigate the complexities of MMM solutions alone. ELIYA's team of experts is here to guide you through your MMM's strategic development and implementation. Our proven track record with top-tier companies ensures that your MMM adaptation is smooth and effective. Reach Out now to discover how we can transform your marketing app.