NoisyAG-News: A Benchmark for Addressing Instance-Dependent Noise in Text Classification

Real-world text classification is challenged by instance-dependent noise (IDN), yet Learning with Noisy Labels (LNL) research predominantly relies on simplistic synthetic models, creating a critical disconnect between academic evaluation and practice. To bridge this gap, we introduce NoisyAG-News, a large-scale benchmark with controllable, real-world IDN. Constructed via a meticulous multi-annotator process on 50,000 samples, our benchmark embodies the genuine cognitive biases inherent in human annotation. Our analysis of this benchmark yields two fundamental insights. First, we reveal why realistic IDN is substantially more destructive than synthetic noise: unlike the initial resistance models show to synthetic errors, it causes immediate learning contamination that leads to a catastrophic, generalizable "Short-Plank Effect." Second, we identify the distinct causal mechanisms of realistic noise sources: a top-down, semantic "Fallback" for human errors versus a bottom-up, feature-driven "Collapse" for LLM biases. NoisyAG-News provides a crucial testbed and a deeper understanding of noise to accelerate the development of LNL algorithms that are genuinely robust to real-world challenges.