LLM-Failure-Cases

# Codatta LLM Failure Cases (Expert Critiques)

## Overview

**Codatta LLM Failure Cases** is a specialized adversarial dataset designed to highlight and analyze scenarios where state-of-the-art Large Language Models (LLMs) produce incorrect, hallucinatory, or logically flawed responses.

This dataset originates from **Codatta's "Airdrop Season 1" campaign**, a crowdsourced data intelligence initiative where participants were tasked with finding prompts that caused leading LLMs to fail. Submissions were rigorously reviewed to ensure objective error and scientifically sound critiques.

**Key Features:**
* **Adversarial Nature:** Represents a "Hard Set" of prompts that have successfully defeated top-tier models.
* **Multi-Model Coverage:** Captures failures from a diverse range of models, including **GPT series, Gemini, DeepSeek, Qwen, Doubao**, and others.
* **Expert Critiques:** Unlike standard QA datasets, each entry includes a rigorous critique that dissects *why* the model failed (e.g., pointing out misapplied physical symmetries or logical fallacies), rather than just providing the correct answer.

## Dataset Contents

The dataset contains structured records of model failures. Each entry consists of the following fields:

* **`submission_id`** (string): Unique identifier for the submission.
* **`question`** (string): The challenging prompt that induced the failure.
* **`domain`** (string): The subject matter of the question (e.g., `science`, `math`, `coding`, `logic`).
* **`model`** (string): The specific LLM that generated the incorrect response (e.g., `gpt-4o`, `deepseek-v2`, `qwen-max`).
* **`model_answer`** (string): The **incorrect** response generated by the AI, capturing the hallucination or reasoning error.
* **`correct_answer`** (string): The **Expert Critique** and correct solution.

**Note on Redundancy:** A single challenging question may appear multiple times if it successfully induced failures across different models. This design allows for cross-model error analysis on identical prompts.

## Key Statistics

* **Total Examples:** 835 failure cases.
* **Languages:** English (`en`) and Chinese (`zh`).
* **Dataset Size:** ~2.6 MB.
* **Models Covered:** GPT series, Gemini, DeepSeek, Qwen, Doubao, and other SOTA models.
* **Data Source:** Community-driven crowdsourcing (Codatta Airdrop Season 1).

## Usage

This dataset is valuable for researchers and developers working on model alignment and robust evaluation.

**Supported Tasks:**
* **Model Evaluation & Red Teaming:** Benchmarking new models against known failure modes of current SOTA models.
* **Multilingual Hallucination Analysis:** Analyzing error patterns in both English and Chinese contexts.
* **DPO (Direct Preference Optimization):** Using the `model_answer` (negative) and `correct_answer` (positive) pairs to align models away from common reasoning pitfalls.

### Usage Example

```python
from datasets import load_dataset

ds = load_dataset("Codatta/llm-failure-cases", split="train")

# Filter for specific model failures
deepseek_failures = ds.filter(lambda x: "deepseek" in x['model'].lower())
print(f"Found {len(deepseek_failures)} failures for DeepSeek models.")
```

## License and Open-Source Details
* **License:** This dataset is released under the **OpenRAIL** license.

# Codatta LLM 失败案例（专家评析）

## 概述
**Codatta LLM 失败案例**是一款专门的对抗性数据集，旨在揭示并分析当前顶尖大语言模型（Large Language Model，LLM）生成错误、幻觉或逻辑缺陷响应的场景。
本数据集源自Codatta发起的“空投季第一期”（Airdrop Season 1）众包数据情报计划，参与者的任务是搜寻可引发主流大语言模型失效的提示词。所有提交内容均经过严格审核，以确保错误案例的客观性与评析的科学性。

## 核心特性
* **对抗属性：** 属于“困难集”（Hard Set），收录了成功绕过顶尖模型的提示词。
* **多模型覆盖：** 涵盖多款不同模型的失效案例，包括GPT系列、Gemini、DeepSeek、Qwen、豆包（Doubao）等。
* **专家评析：** 与标准问答数据集不同，每条数据均包含严谨的评析，深入剖析模型失效的根源（例如错误应用的物理对称性或逻辑谬误），而非仅提供正确答案。

## 数据集内容
本数据集包含结构化的模型失效记录，每条数据包含以下字段：
* **`submission_id`**（字符串）：提交项的唯一标识符。
* **`question`**（字符串）：引发模型失效的挑战性提示词。
* **`domain`**（字符串）：问题所属的学科领域（例如`science`（科学）、`math`（数学）、`coding`（编程）、`logic`（逻辑））。
* **`model`**（字符串）：生成错误响应的具体大语言模型（例如`gpt-4o`、`deepseek-v2`、`qwen-max`）。
* **`model_answer`**（字符串）：AI生成的**错误响应**，包含幻觉内容或推理错误。
* **`correct_answer`**（字符串）：**专家评析与正确解决方案**。

**冗余说明：** 若某一挑战性提示词可在多款模型上引发失效，则该提示词可能多次出现。此设计支持针对同一提示词开展跨模型错误分析。

## 核心统计数据
* **总样本数：** 835个失效案例。
* **支持语言：** 英语（`en`）与中文（`zh`）。
* **数据集大小：** 约2.6 MB。
* **覆盖模型：** GPT系列、Gemini、DeepSeek、Qwen、豆包（Doubao）及其他顶尖模型。
* **数据来源：** 社区驱动的众包模式（Codatta空投季第一期）。

## 应用场景
本数据集对致力于模型对齐与鲁棒性评估的研究者与开发者具有重要价值。
**支持任务：**
* **模型评估与红队测试：** 基于当前顶尖模型已知的失效模式，对新模型开展基准测试。
* **多语言幻觉分析：** 分析英语与中文语境下的错误模式。
* **DPO（直接偏好优化，Direct Preference Optimization）：** 利用`model_answer`（负样本）与`correct_answer`（正样本）对，使模型规避常见的推理陷阱。

### 使用示例
python
from datasets import load_dataset

ds = load_dataset("Codatta/llm-failure-cases", split="train")

# 筛选特定模型的失效案例
deepseek_failures = ds.filter(lambda x: "deepseek" in x['model'].lower())
print(f"Found {len(deepseek_failures)} failures for DeepSeek models.")

## 授权与开源详情
* **授权协议：** 本数据集采用**OpenRAIL**协议发布。