Article-level image suggestions evaluation (<strong>ALISE</strong>) dataset

Article-level image suggestions (<strong>ALIS</strong>, reads <em>alice</em>) is a distributed computing system that recommends images for Wikipedia articles that don't have one [1].<br> This publication contains roughly <strong>3,800</strong> human ratings made against ALIS output in multiple Wikipedia language editions. Evaluation task Data was collected through an evaluation tool [2], with code available at [3]. Given a language, the user is shown a random Wikipedia article and an image suggested by the system; they are then asked to rate the relevance of the image by clicking on either the <em>Good, Okay, Bad,</em> or <em>Unsure</em> button. The user is also brought to judge whether the image is not suitable for any reason via the <em>It's ok, It's unsuitable</em>, or <em>Unsure</em> button. Content The archive holds 2 tab-separated-values (TSV) text files: <em>evaluation_dataset.tsv</em> contains the evaluation data; <em>unillustrated_articles.tsv</em> keeps track of unillustrated Wikipedia articles. Evaluation dataset headers <strong>id</strong> (integer) - identifier used for internal storage; <strong>unillustratedArticleId</strong> (integer) - identifier of the unillustrated Wikipedia article; <strong>resultFilePage</strong> (string) - Wikimedia Commons image file name. Prepend <em>https://commons.wikimedia.org/wiki/</em> to form a valid Commons URL; <strong>resultImageUrl</strong> (string) - Wikimedia Commons thumbnail URL; <strong>source</strong> (string) - suggestion source. <strong>ms</strong> = MediaSearch; <strong>ima</strong> = ALIS prototype algorithm. See [4] and [5] respectively for more details; <strong>confidence_class</strong> (string) - shallow degree of suggestion confidence. Either <strong>low</strong>, <strong>medium</strong>, or <strong>high</strong>; <strong>rating</strong> (integer) - human image relevance rating. <strong>1</strong> = good; <strong>0</strong> = okay; <strong>-1</strong> = bad; <strong>sensitive</strong> (integer) - human image suitability rating. <strong>0</strong> = it's okay; <strong>1</strong> = it's unsuitable; <strong>-1</strong> = unsure; <strong>viewCount</strong> (integer) - number of times the suggestion was seen by evaluators. Example 7357 1827 File:Cuphea_cyanea_strybing.jpg https://upload.wikimedia.org/wikipedia/commons/thumb/1/17/Cuphea_cyanea_strybing.jpg/800px-Cuphea_cyanea_strybing.jpg ima high 1 0 1 Unillustrated articles headers <strong>id</strong> (integer) - identifier used for internal storage. Maps to <strong>unillustratedArticleId</strong> in the evaluation data; <strong>langCode</strong> (string) - Wikipedia language code; <strong>pageTitle</strong> (string) - Wikipedia article title; <strong>unsuitableArticleType</strong> (integer) - whether the Wikipedia article is suitable for receiving image suggestions. <strong>0</strong> = suitable; <strong>1</strong> = not suitable; Example 1827 vi Cuphea_cyanea 0 References [1] https://www.mediawiki.org/wiki/Structured_Data_Across_Wikimedia/Image_Suggestions/Data_Pipeline [2] https://image-recommendation-test.toolforge.org/ [3] https://github.com/cormacparle/media-search-signal-test/tree/master/public_html [4] https://www.mediawiki.org/wiki/Help:MediaSearch [5] https://www.mediawiki.org/wiki/Structured_Data_Across_Wikimedia/Image_Suggestions/Data_Pipeline#How_it_works