Framenet Semantic Frame Disambiguation With Crowdtruth

This repository contains a ground truth corpus for semantic frame disambiguation, acquired with crowdsourcing and processed with <strong>CrowdTruth</strong> metrics that capture ambiguity in annotations by measuring inter-annotator disagreement. The dataset contains annotations for 433 sentence-word pairs from the FrameNet corpus v.1.7, with each sentence-word pair annotated for frame disambiguation by 15 workers. The crowdsourced data was collected from Amazon Mechanical Turk. The corpus has been referenced in the following paper: Anca Dumitrache, Lora Aroyo and Chris Welty: <strong>Capturing and Interpreting Ambiguity in Crowdsourcing Frame Disambiguation</strong>. HCOMP 2018. To replicate the data processing from the paper, use the Jupyter Notebook file <code>CrowdTruth metrics.ipynb</code>. It requires the installation of the CrowdTruth metrics Python package (v &gt;= 2.0). The data aggregated with CrowdTruth metrics is available in folder <code>data/output/</code> The raw crowdsourcing data is available in folder <code>data/input/</code> If you find this data useful in your research, please consider citing: <pre><code>@inproceedings{dumitrache2018frames, Author = {Anca Dumitrache and Lora Aroyo and Chris Welty}, Title = {Capturing Ambiguity in Crowdsourcing Frame Disambiguation}, Booktitle = {The sixth AAAI Conference on Human Computation and Crowdsourcing}, Year = {2018} } </code></pre>