Factorial Designs for Online Experiments

Online experiments and specifically A/B testing are commonly used to identify whether a proposed change to a web page is in fact an effective one. This study focuses on basic settings in which a binary outcome is obtained from each user who visits the website and the probability of a response may be affected by numerous factors. We use Bayesian probit regression to model the factor effects and combine elements from traditional two-level factorial experiments and multiarmed bandits to construct sequential designs that embed attractive features of estimation and exploitation.