MultiChallenge

# MultiChallenge MultiChallenge is a benchmark for evaluating large language models on multi-turn conversations. It tests whether models can maintain context, follow instructions, and reason correctly across extended dialogues. ## Dataset Description - **Size:** 266 examples - **Split:** test - **Language:** English ## Evaluation Axes | Axis | Description | |------|-------------| | `INFERENCE_MEMORY` | Tests whether the model can recall and reason over information from earlier turns | | `INSTRUCTION_RETENTION` | Tests whether the model continues to follow instructions given in earlier turns | | `SELF_COHERENCE` | Tests whether the model's responses remain internally consistent across turns | | `RELIABLE_VERSION_EDITING` | Tests whether the model can reliably edit/update content across turns | ## Dataset Structure Each example contains: - **`question_id`**: Unique identifier for the question - **`axis`**: The evaluation axis being tested - **`conversation`**: List of conversation turns (role + content) - **`target_question`**: The evaluation question to assess model performance - **`pass_criteria`**: Expected answer for the evaluation question - **`num_turns`**: Number of turns in the conversation ## Usage ```python from datasets import load_dataset dataset = load_dataset("ScaleAI/MultiChallenge", split="test") print(dataset[0]) ```