Crowdsourced dataset to study the generation and impact of text highlighting in classification tasks

Here we present the datasets derived from our experiments on using crowdsourcing for document classification tasks. These experiments resemble a two-step process that first highlights excerpts from the text and then leverage these to workers for classification. Thus our experiments groups into highlighting generation and classification. For generating highlights, we leverage crowdsourcing and automatic approaches such us extractive summarization and question answering models. For our classification experiments, we consider documents from two different domains: systematic literature reviews and amazon product reviews. Specifically, we study how highlighting text passages could aid workers in judging the relevance of a document given an input question. We spec these datasets to benefit not only to study these particular problem domains but a broader set of classification problems where individual judgments from workers are scarce.<br>