Crowdsourced datasets to study the generation and impact of text highlighting in classification tasks

Here we present the datasets derived from our experiments on using crowdsourcing for document classification tasks. These experiments resemble a two-step process that first highlights excerpts from the text and then leverage these to workers for classification. Thus our experiments groups into highlighting generation and classification. For generating highlights, we leverage crowdsourcing and automatic approaches such us extractive summarization and question answering models. For our classification experiments, we consider documents from two different domains: systematic literature reviews and amazon product reviews. Specifically, we study how highlighting text passages could aid workers in judging the relevance of a document given an input question. We spec these datasets to benefit not only to study these particular problem domains but a broader set of classification problems where individual judgments from workers are scarce.<br><br>In a nutshell, the datasets represent two kinds of tasks:<br>- classification tasks with highlighting support.- highlighting tasks, where the workers highlight evidence.<br><b>Classification tasks</b><br>In this task, workers classified documents based on a given predicate. <br><b>classification tasks using crowdsourced highlights</b><br>Files:<i>- classification_amazon-crowd-highlights.csv</i><i>- classification_oa-crowd-highlights.csv</i><i>- classification_tech-crowd-highlights.csv</i><i>- classification_tech-3x12-crowd-highlights.csv</i><i>- classification_tech-6x6-crowd-highlights.csv</i><br><b>classification tasks using ML-generated highlights</b><br>Files:<i>- classification_amazon-ML-highlights.csv</i><i>- classification_oa-ML-highlights.csv</i><i>- classification_tech-ML-highlights.csv</i><br><br><b>Highlighting tasks</b><br><b>crowdsourced highlights</b><br><br>In this task, workers highlighted excerpts from documents that are relevant to a given predicate, to support future classification tasks.<br>File: <i>crowdsourced_highlights.csv.</i><br><i><br></i>The file contains one line per highlight (generated by one worker); the column that holds the highlighted fragment(s) is <i>highlighted_text</i>. The <i>highlighted_text</i> is a "list of lists" (Python syntax), so iterating over this list will give you the text fragment generated by one worker. Also, the <i>experiment</i> column indicates domain + task design. So, to get the highlights used in the classification experiments, use the rows that end with <i>"-highlight"</i>.<br><br><b>ML-generated highlights</b><br>We also consider automatic approaches to generate text highlights — specifically, extractive summarization and question-answering models.<br><br>File: <i>ml_highlights.csv.</i>