Funnel metrics use Bayesian statistics with a beta model to evaluate the win probabilities and credible intervals. Read the statistics overview if you haven't already.
What is a beta model?
Imagine you run a pizza shop and want to know if customers say "yes" to adding pineapple. Some customers will say yes, others will say no. Knowing what percentage of customers want pineapple on their pizza helps you decide how much to order and what options to offer.
The beta model is a statistical approach that's great for analyzing proportions or probabilities. It uses a beta distribution to model the uncertainty in conversion rates and helps us understand:
- Our best estimate of the true probability that a customer will say "yes" to adding pineapple (vs. the probability we observe).
- How certain we are about that estimate.
For example, if:
- Only 2 out of 4 customers (50%) say yes, the beta distribution will be wide, indicating high uncertainty.
- 150 out of 300 customers (50%) say yes, the beta distribution will be narrow, showing we're more confident about that 50% rate.
So when we say we're using a beta model for funnel metrics, we're:
- Using the beta distribution to model conversion rates between 0% and 100%.
- Getting more confident in our estimates as we collect more data.
One more thing worth noting: Bayesian inference starts with an initial guess that then gets updated as more data comes in. Our model uses a "minimally informative prior" of ALPHA_PRIOR = 1 and BETA_PRIOR = 1, which is like starting with a blank slate instead of making an upfront assumption about the results.
Win probabilities
The win probability tells you how likely it is that a given variant has the highest conversion rate compared to all other variants. It helps you determine whether the metric shows a statistically significant real effect vs. simply random chance.
Let's say you're testing a new way of presenting pineapple on the website and have these results:
- Control (current design): 100 pineapple orders from 1000 customers (10% acceptance)
- Test (suggesting pineapple with a photo): 150 pineapple orders from 1000 customers (15% acceptance)
To calculate the win probabilities, our methodology will:
Model each variant's conversion rate using a beta distribution:
- Control: Beta(100 + ALPHA_PRIOR, 900 + BETA_PRIOR)
- Test: Beta(150 + ALPHA_PRIOR, 850 + BETA_PRIOR)
Take 10,000 random samples from each distribution.
Check which variant had the higher conversion rate for each sample.
Calculate the final win probabilities:
- Control wins in 40 out of 10,000 samples = 0.4% probability
- Test wins in 9,960 out of 10,000 samples = 99.6% probability
These results tell us we can be 99.6% confident that showing photos of pineapple pizza performs better than the current design.
Credible intervals
A credible interval tells you the range where the true conversion rate lies with 95% probability. This is different than a confidence interval, which describes how often such intervals would contain the true rate if you repeated the experiment many times (not a direct probability statement about where the rate lies).
For example, if you have these results:
- Control (current design): 100 pineapple orders from 1000 customers (10% acceptance)
- Test (suggesting pineapple with a photo): 150 pineapple orders from 1000 customers (15% acceptance)
To calculate the credible intervals, our methodology will:
Create a beta distribution for each variant:
- Control: Beta(100 + ALPHA_PRIOR, 900 + BETA_PRIOR)
- Test: Beta(150 + ALPHA_PRIOR, 850 + BETA_PRIOR)
Find the 2.5th and 97.5% percentiles of each distribution:
- Control: [8.3%, 12%] = "You can be 95% confident the true conversion rate is between 8.3% and 12.0%"
- Test: [12.9%, 17.3%] = "You can be 95% confident the true conversion rate is between 12.9% and 17.3%"
Since these intervals don't overlap, you can be quite confident that the test variant performs better than the control. The intervals will become narrower as you collect more data, reflecting your increasing certainty about the true conversion rates.