eval-IFBench-results

# IFBench Evaluation Results This dataset contains evaluation results for various language models on [IFBench](https://github.com/shisa-ai/IFBench), a challenging benchmark for precise instruction following. **Naming Convention**: This repo follows the `eval-{EVAL}-{type}` schema for organizing evaluation datasets. Related repos: - `eval-IFBench-results` - Model evaluation outputs (this repo) - `eval-IFBench-prompts` - Test prompts/questions (if separated) ## Dataset Structure Results are organized by model name: ``` {model-name}/ ├── responses_{model-name}.jsonl # Model responses to IFBench prompts ├── eval_results_strict.jsonl # Strict evaluation results └── eval_results_loose.jsonl # Loose evaluation results ``` ### File Formats **responses.jsonl** - Model outputs ```json {"prompt": "...", "response": "..."} ``` **eval_results_*.jsonl** - Evaluation results ```json { "prompt": "...", "response": "...", "follow_all_instructions": true/false, "follow_instruction_list": [true, false, ...], "instruction_id_list": ["constraint:id", ...] } ``` ## Evaluated Models | Model | Strict Accuracy | Loose Accuracy | Date | |-------|-----------------|----------------|------| | *Results will be added as evaluations complete* | | | | ## Usage ```python from datasets import load_dataset # Load all results ds = load_dataset("shisa-ai/eval-IFBench-results") # Or load specific model results import json from huggingface_hub import hf_hub_download path = hf_hub_download( repo_id="shisa-ai/eval-IFBench-results", filename="meta-llama--Llama-3.3-70B-Instruct/eval_results_loose.jsonl", repo_type="dataset" ) with open(path) as f: results = [json.loads(line) for line in f] ``` ## Contributing Results We welcome community contributions! To submit evaluation results: 1. **Fork this repository** on HuggingFace 2. **Run IFBench evaluation** using the [official code](https://github.com/shisa-ai/IFBench) 3. **Add your results** in a folder named `{org}--{model-name}/` 4. **Open a Pull Request** via the Community tab ### Required files: - `responses_{model-name}.jsonl` - Your model's responses - `eval_results_strict.jsonl` - Strict evaluation output - `eval_results_loose.jsonl` - Loose evaluation output ### Naming convention: Use HuggingFace model ID format: `organization--model-name` Example: `meta-llama--Llama-3.3-70B-Instruct` ## Citation If you use these results, please cite the IFBench paper: ```bibtex @misc{pyatkin2025generalizing, title={Generalizing Verifiable Instruction Following}, author={Valentina Pyatkin and Saumya Malik and Victoria Graf and Hamish Ivison and Shengyi Huang and Pradeep Dasigi and Nathan Lambert and Hannaneh Hajishirzi}, year={2025}, journal={Advances in Neural Information Processing Systems}, volume={38} } ``` ## License Results are released under ODC-BY-1.0, consistent with the original IFBench dataset license.