I’m working on Bayesian AB Tests for a showcase project.
Then I decided to do a quick memo on some principles regarding Bayesian AB Tests
Principles of Bayesian AB Tests
- You have beliefs (a priori).
- You do an experiment to test these beliefs.
- You get a result (a posteriori).
- You combine both to revise your first beliefs.
Example: you want to change the background colour of your website: from blue to green. You have 2 variants: blue (the old colour) and green (the new one) that you want to test.
You’ve done some research and know that in your industry, green is a colour generates 16% of conversion whereas blue generates 10% in your company.
These 16% and 10% are your prior beliefs. The experiment gives you some results. And the combination of both is the revision of your former beliefs: your posterior beliefs.
Set your prior beliefs as a beta distribution
To set a distribution as prior beliefs, you can use a Beta distribution.
The Beta distribution has two parameters: alpha => the number of successes beta => the number of failures
If you think that your new color will give 16% of conversion, you can set your beta distribution as: Beta(16, 84).
For a total of 100 trials, you get 16 successes and 86 failures.
Modulate your prior beliefs
The higher the numbers in a beta distribution, the more confident you are. High numbers will decrease the variance of your curve whereas low numbers flatten your curve.
- Beta(16, 84) if you’re confident enough
- Beta(8, 42) if you’re neutral
- Beta(4, 21) if you’re sceptical
The ratio is the same (16%), but not the curve and then not the strength of your beliefs.