AB testing: Learnings from Meta

From launching entirely new products to fine-tuning the smallest UI changes, AB testing is the go-to tool for making data-informed decisions at companies big and small.

But when it comes to Meta, with billions of users worldwide relying on our platforms to connect, communicate, and run their businesses, the scale & complexity of AB testing takes a whole different dimension.

In this article, I'll share my own experiences working on AB testing at Meta, highlighting the unique challenges, strategies, and nuances involved in conducting experiments at the world's largest social fabric.

Whether you are a product manager, engineer, data scientist, or simply interested in the intersection of data and product growth, you will gain valuable insights from this exploration. Read on!

Why is AB testing so important at Meta?

Before we dive into specifics, let’s take a moment to understand why AB testing is so important at Meta:

Man holding a sign that reads: "I need more data"

With massive scale comes massive responsibility:

We are the biggest social fabric connecting the world with billions of people across the world relying on our products to connect with friends & family, run online businesses, and discover content they enjoy.

However, at this scale, even seemingly minor changes can have far-reaching consequences.

To mitigate this risk, we rely on AB testing to rigorously evaluate the impact of potential changes before rolling them out more broadly. By doing so, we can protect the user experience and prevent unintended changes from affecting our global community.

Building for a diverse user base presents both an opportunity and a challenge:

At Meta, our global user base is incredibly diverse, comprising people from various demographics who utilize our platforms for different purposes. This diversity presents both an opportunity and a challenge, as we strive to create products that are relevant and effective for each unique audience segment.

To achieve this goal, AB testing plays a critical role in helping us test, learn, and iterate on product features that cater to the distinct needs of our users worldwide.

By leveraging AB testing, we can ensure that our products meet the diverse requirements of our global user base, ultimately enhancing their overall experience.

Our product ecosystem is interconnected and complex:

Our product portfolio is a complex ecosystem of interconnected platforms, including Facebook, Instagram, WhatsApp, and Oculus, each with numerous features that interact and influence one another.

This intricate web of relationships means that even small changes to one feature can have unintended consequences on others, potentially cannibalizing usage or disrupting the delicate balance of our ecosystem.

For instance, a new feature designed to boost reels production might inadvertently lead to a decline in stories production. To mitigate these risks, AB testing becomes an essential tool for isolating and evaluating specific changes within our ecosystem, ensuring that improvements in one area do not come at the expense of others.

We need to move fast and innovate often:

At Meta, we must move quickly to stay ahead of the curve and meet the ever-changing needs of billions of users worldwide.

With user expectations constantly shifting and competition intensifying, AB testing is key to our success, allowing us to rapidly validate ideas, iterate on user feedback, and launch products that delight – all while ensuring a seamless user experience and protecting the integrity of our ecosystem.

AB testing to drive product goals

AB testing is a trusted ally, as we take the product through different stages of growth and optimization. Below, I'll share valuable insights gained from using AB testing at Meta to achieve distinct product objectives, at different stages in the product lifecycle:

Image of a man in a suit with a superhero cape and mast with text reading: "help me AB testing you are my only hope"

Product goal: Attain product-market fit for a 0 to 1 product

For a new product, attaining product-market fit (PMF) is key for success and also the hardest step since the idea is fresh out of the oven with little to no prior data available.

Herein, AB testing plays a crucial role in validating early assumptions, collecting user feedback and data to inform product direction, to ultimately attain PMF.

Breaking this journey to steps:

Step 1: Early stage validation

Even the best of ideas need to be validated before they are put to the test. At Meta, we take a structured approach to idea validation by conducting user research, industry & competitor analysis to answer three key questions: Who, What, How much

Who: Which segment of users do we solve for?
What: What problem do we intend to solve for these users? How severe is the problem and is the problem largely unsolved, or does a workaround exist?
How much: What is the opportunity size, i.e. how much impact can we drive by solving this problem?

If the idea is viable based on the above analysis, we move on to the next step – define MVP.

💡

At Meta, we hold a high bar before putting any idea to AB test primarily because the experience of billions of users is at stake and there are both ethical, and legal risks associated with testing.

Hence, we invest a lot of time upfront in validating ideas - putting only the high-confidence ideas in the AB test.

Step 2: Define MVP and success metrics

At Meta, when defining our MVP, we optimize for impact, simplicity, and speed. We conduct user interviews to identify core features that resonate with users, then prioritize MVP features based on their potential impact and engineering cost, ensuring a quick and effective path to market.

Then to measure the success of our MVP holistically, we use a comprehensive set of metrics that assess adoption, engagement & retention. This approach eliminates novelty bias and detects potential issues in the user journey, providing a more accurate understanding of the product's performance:

Adoption: Establish a baseline adoption rate within your Total Addressable Market (TAM) by referencing similar products or industry benchmarks. For early-stage PMF, aim for 1-5% adoption, indicating initial traction and validation with a small user group
Engagement rate: Measure the depth of user interaction with core features, such as frequency of use
Funnel analysis: Track users' progression through the product funnel, measuring conversion rates at each step
Retention: Are users coming back to use your product regularly, indicating the product is unlocking value for users?

Define your hypotheses for MVP using this formula: ,“Users will find value in features <X,Y> to solve the problem <Z>, which in turn will drive metrics <a,b,c>”

💡

At Meta, we care about driving topline impact in terms of user sessions, time spent, etc. as that indicates real user value. However, topline metrics are hard to move in AB tests and sometimes take multiple feature rollouts, over the course of time, to detect movement.

Therefore, to measure product success in the short term, we establish local product metrics that are movable in AB tests and ladder to topline metrics. Furthermore, we establish long-term holdouts to measure topline impact from cumulative product launches over time.

Step 3: Run AB tests - Measure, learn, and iterate to PMF

With a clear hypothesis in place, we bring the product to AB testing and iterate to achieve PMF using this approach:

We test multiple variants of MVP to see what resonates the most with users. Based on test results, we revise the hypotheses and continue to iterate to refine the product
We always, always supplement our learnings with qualitative feedback through in-product surveys and user interviews. We nurture early adopters, as they provide crucial feedback, evangelize the product, and help shape its direction
To detect PMF, we monitor product usage patterns over time. We know we have attained PMF when the product shows a baseline adoption rate (as defined in success metrics) sustained over time. Adoption here means users engage with the product in a meaningful way and come back to use it consistently (aka retention).

💡

When shipping features based on AB test results, always keep an eye out for long tail effects.

Particularly at Meta, due to the interconnected nature of features and platforms, any change can have a cascading effect, which is not always detectable in AB Tests.

To account for this, we ship features with backtests to monitor long-term impact and might unship if the impact vanishes, OR we start seeing regressions in guardrails.

When testing new products, you might encounter not statistically significant results due to low usage (despite testing on a big sample size).

To overcome this challenge at Meta, we supplement our understanding with offline analysis using queries to gain insights into funnel usage and other key metrics to inform future iterations

Product goal: Grow the product

Once you have attained PMF, it’s time to grow your product. No matter what strategy you choose to grow your product, AB testing can serve as a powerful aid in the process:

Strategy 1: Deepen engagement with existing users

Reduce friction points in the product (optimize product funnel) by:

Deepening engagement by identifying and eliminating friction points within the product. Analyze the product funnel, identify drop-off points, and conduct user interviews to understand the reason for drop-offs.
Next, brainstorm ideas to reduce friction and put them to an AB test to see if that moves the needle
Move beyond MVP and deepen engagement by introducing new features that delight & further elevate the user experience
Gamify – incorporate game-like elements (badges, leaderboards, rewards etc.) to increase user engagement by making the experience more enjoyable. At Meta, we gamified the creation experience for creators, which was a big hit with creators.

Next, nudge your users.

Users have multiple platforms competing for their time. Notifications are a great way to remind users and bring them back to the product for more engagement.

Targeting is particularly important for notifications, so ensure you, reach the right users, with the right content, at the right time, on surfaces they are most active on. AB tests can be used to test multiple targeting variants for notifications and ship the winning experience that drives engagement without negatively impacting user experience.

💡

In the product growth stage, we are scrappy - we move fast, we pivot quickly. AB tests play a crucial role in the process, helping us quickly validate ideas, with minimal resources and maximal impact

With notifications, it is extremely important to protect the user experience and check for spamminess. To address this, we conduct product surveys and set up strong guardrails to measure regressions in user experience.

Users have limited time to spend on social media. As a result, when testing incremental features to deepen engagement, we could end up cannibalizing other products.

At Meta, we set up guardrails to detect cannibalization and only ship features that are incremental to the user experience overall.

Strategy 2: Acquire new users

Identify new customer segments to grow your product. This could involve expanding to new countries or regions and localizing the product to meet the unique needs of the market.

At this stage, rely heavily on user interviews to understand the needs and identify the right features to deliver value to new markets

Next, invest in awareness levers to introduce the product to new users. Marketing campaigns and in-product notifications can be a great way to drive awareness. At Meta, we mostly rely on in-product awareness levers, as most users are already active on the platform using other features.

💡

Meta's platforms serve a global audience with different values, norms, and sensitivities. We design AB tests, keeping these cultural sensitivities in mind, and protect our users from unintended cultural, social, or political implications

Final thoughts

To conclude, AB testing is a powerful tool to advance a product through different stages in its lifecycle – it’s a mindset that’s imperative to innovate and stay ahead of the curve in the tech world.

By harnessing its power, you can be confident in crafting products that resonate with users, drive business success, and leave a lasting impact on the world!