DeepPrune

# 📁 DeepPrune Datasets This repository contains the datasets used in the **DeepPrune** experiments, organized into three categories: **fine-tuning**, **offline evaluation**, and **online evaluation**. ## 🛠️ Fine-tuning Datasets Located in the `finetune_datasets/` directory, these datasets are formatted for use with **Llama-Factory** and include: - `train.jsonl` – Training data - `test.jsonl` – Evaluation data Each line in these `.jsonl` files is a JSON object with the following fields: ```json { "instruction": "It's like a system prompt or task description", "input": "Two truncated answers to be checked whether their answers are identical", "output": "The expected model response: identical/ not identical" } ``` These datasets are used to fine-tune base models before applying the DeepPrune pruning strategy. --- ## 📊 Offline Evaluation Datasets In the `offline_datasets/` directory, we provide model-generated responses from the following models on a shared set of problems: - `DeepSeek-R1-Distill-llama-8b` - `glm-4.5-air` - `Qwen3-4B-Thinking-2507` - `QwQ-32B` These outputs are used to **evaluate the performance of models after fine-tuning**. --- ## 🌐 Online Evaluation Datasets The `online/` directory contains datasets collected through **active querying** of large language models. Specifically: - For each problem, we gathered **512 model-generated answers** from: - `DeepSeek-R1-0528-Qwen3-8B` - `gpt-oss-20b` - `Qwen3-32B` Each JSON file in this folder includes the following fields: ```json { "problem": "The original question or task", "answer": "The model's generated response", "true_answer": "The ground-truth or reference answer" } ``` These datasets are used to **empirically validate the effectiveness of DeepPrune** in real-world, dynamic settings—measuring how pruning impacts output quality under diverse sampling conditions.