CLEVRER-Humans: Describing physical and causal events the human way

Building machines that can reason about physical events and their causal
relationships is crucial for flexible interaction with the physical world.
However, most existing physical and causal reasoning benchmarks are
exclusively based on synthetically generated events and synthetic natural
language descriptions of causal relationships. This design brings up two
issues. First, there is a lack of diversity in both event types and
natural language descriptions; second, causal relationships based on
manually-defined heuristics are different from human judgments. To address
both shortcomings, we present the CLEVRER-Humans benchmark, a video
reasoning dataset for causal judgment of physical events with human
labels. We employ two techniques to improve data collection efficiency:
first, a novel iterative event cloze task to elicit a new representation
of events in videos, which we term Causal Event Graphs (CEGs); second, a
data augmentation technique based on neural language generative models. We
convert the collected CEGs into questions and answers to be consistent
with prior work. Finally, we study a collection of baseline approaches for
CLEVRER-Humans question-answering, highlighting the great challenges set
forth by our benchmark.