My Climate Copilot User Study Annotations

A collection of 357 annotations from 13 climate experts for Climate Adaptation NLP. It contains evaluations of Large Language Model (LLM) capability on climate questions as well as the raw responses from LLMs.\n\nThis collection is useful for benchmarking models on their generation performance to expert climate adaptation questions and also measuring their evaluation performance in comparison to experts.\nLineage: 3 Language Models (LLMs) were used to generate responses to 50 questions posed by climate experts. LLMs generated responses using climate data/scientific literature, and without, amounting to 300 unique responses. \n\n13 Experts were asked to annotate responses from the different with a set of 7 criteria (annotation guidelines included), which were created by experts. 57 responses were annotated twice by a different expert to measure inter-annotator agreement. \n\nEvaluation criteria can be found in the user testing guide PDF. Each sub-criterion is labelled in lexicographic order (i.e., structure_a refers to the first sub-criterion in the user testing guide for the structure criteria). A score of 1 indicates that an expert believes the response meets that sub-criteria, while a score of 0 indicates the response failed to meet that sub-criteria.