Rerandomization Algorithms for Optimal Designs of Network A/B Tests

A/B testing is an effective method to assess the potential impact of two treatments. For A/B tests conducted by IT companies like Meta and LinkedIn, the test users can be connected and form a social network. Users’ responses may be influenced by their network connections, and the quality of the treatment estimator of an A/B test depends on how the two treatments are allocated across different users in the network. This article investigates optimal design criteria based on some commonly used outcome models, under assumptions of network-correlated outcomes or network interference. We demonstrate that the optimal design criteria under these network assumptions depend on several key statistics of the random design vector. We propose a framework to develop algorithms that generate rerandomization designs meeting the required conditions of those statistics under a specific assumption. Asymptotic distributions of these statistics are derived to guide the specification of parameters in the algorithms. We validate the proposed algorithms using both synthetic and real-world networks.

A/B测试是评估两种处理方案潜在影响的有效方法。对于Meta、LinkedIn等IT企业开展的A/B测试而言，参与测试的用户之间可能存在关联并形成社交网络。用户的响应可能受到其网络连接的影响，而A/B测试中处理效应估计量的质量则取决于两种处理方案在网络内不同用户间的分配方式。本文在网络相关结果（network-correlated outcomes）或网络干扰（network interference）的假设下，基于若干常用的结果模型探讨最优设计准则。我们证明，在这些网络假设下，最优设计准则依赖于随机设计向量的若干关键统计量。我们提出一个框架，用于开发能够生成满足特定假设下这些统计量所需条件的重随机化设计（rerandomization designs）的算法。我们推导了这些统计量的渐近分布（asymptotic distributions），以指导算法中参数的设定。我们利用合成网络和真实世界网络对所提出的算法进行了验证。