False conclusions in online economic experiments

Bias caused by endogenous subject attrition endangers the internal validity of economic experiments. Non-completion is common in experiments conducted online, but often not addressed in design, analysis, and reporting. We survey the literature and find the large majority of papers using online experiments do not report attrition rates. We demonstrate the risk of attrition bias with two experiments reaching false conclusions; the first obviously false and the second plausible but false. We then test six proposed attrition reduction methods and find five to be ineffective. Use of a warm-up task was highly effective reducing attrition by more than two-thirds.