When LLMs Disagree with Human Experts: Understanding LLM Annotation Failures in Nutrition Misinformation using Hierarchical Error Analysis

This repository contains dataset, code, and analysis for evaluating large language models as annotators of nutrition misinformation on Instagram. We introduce a hierarchical error taxonomy to analyze LLM misclassification patterns across multiple open-source models, grounded in expert nutritionist annotations based on the U.S. Dietary Guidelines (2020–2025). The dataset includes 169 Instagram captions related to seed oils, annotated by domain experts, along with LLM predictions and error analyses.