Section-level image suggestions evaluation (<strong>SLISE</strong>) datasets

Section-level image suggestions (<strong>SLIS</strong>, reads <em>slice</em>) is a distributed computing system that recommends images for Wikipedia article sections [1]. <br> This publication contains roughly <strong>9,000</strong> human ratings made against SLIS output in multiple Wikipedia language editions. <br> Evaluation task Data was collected through an evaluation tool [2], with code available at [3]. Given a language, the user is shown a random Wikipedia article section together with an image suggested by the system and is asked to judge the relevance of the image by clicking on one of the following buttons: <em>Good, Bad, Unsure, This section shouldn't have an image, The image is offensive.</em> <br> Content Each dataset is a tab-separated-values (TSV) text file that corresponds to an evaluation round. <br> Headers <strong>id</strong> (integer) - identifier used for internal storage; <strong>dataset_id</strong> (integer) - evaluation round identifier; <strong>suggestion_origin</strong> (string) - algorithm component that generated the suggestion. See [4] for more details; <strong>wiki_db</strong> (string) - Wikipedia language edition; <strong>target_qid</strong> (string) - Wikidata identifier of the Wikipedia article; <strong>target_title</strong> (string) - Wikipedia article title; <strong>target_section_heading</strong> (string) - Wikipedia article section title; <strong>suggested_image</strong> (string) - Wikimedia Commons image file name. Prepend <em>https://commons.wikimedia.org/wiki/File:</em> to form a valid Commons URL; <strong>topic_qid</strong> (string) - Wikidata identifier of the wikilink used to generate the suggestion. See [5] for more details; <strong>topic_score</strong> (float) - wikilink relevance score. See [6] for more details; <strong>score_level</strong> (string) - wikilink relevance score level, either <em>article</em> or <em>section</em>. See [6] for more details; <strong>origin_wikis</strong> (list) - Wikipedia language editions used to generate the suggestion. See [7] for more details; <strong>confidence</strong> (integer) - image suggestion confidence score. See [8] for more details; <strong>rating</strong> (integer) - human rating. <strong>1</strong> = good; <strong>0</strong> = bad; <strong>-4</strong> = unsure; <strong>-1</strong> = this section shouldn't have an image; <strong>-5</strong> = the image is offensive; <strong>ts</strong> (date) - when the rating was submitted. <br> [1] https://www.mediawiki.org/wiki/Structured_Data_Across_Wikimedia/Section-level_Image_Suggestions/Data_Pipeline [2] https://section-image-suggestions-test.toolforge.org/ [3] https://gitlab.wikimedia.org/toolforge-repos/section-image-suggestions-test [4] https://www.mediawiki.org/wiki/Structured_Data_Across_Wikimedia/Section-level_Image_Suggestions/Data_Pipeline#How_it_works [5] https://www.mediawiki.org/wiki/Structured_Data_Across_Wikimedia/Section_Topics/Data_Pipeline [6] https://www.mediawiki.org/wiki/Structured_Data_Across_Wikimedia/Section_Topics/Data_Pipeline#Relevance_score [7] https://www.mediawiki.org/wiki/Structured_Data_Across_Wikimedia/Section-level_Image_Suggestions/Data_Pipeline#Section_alignment [8] https://www.mediawiki.org/wiki/Structured_Data_Across_Wikimedia/Section-level_Image_Suggestions/Data_Pipeline#Confidence_scores