distilabel-internal-testing/prometheus-bench-critique

---
dataset_info:
features:
- name: idx
dtype: int64
- name: hashid
dtype: string
- name: response_source
dtype: string
- name: category
dtype: string
- name: criteria
dtype: string
- name: base_dataset
dtype: string
- name: instruction
dtype: string
- name: response
dtype: string
splits:
- name: test
num_bytes: 4172797
num_examples: 2309
download_size: 1845258
dataset_size: 4172797
configs:
- config_name: default
data_files:
- split: test
path: data/test-*
---

## Description

This is the base dataset to create a benchmark for critique models, starting from the datasets used to benchmark Prometheus models.

## Generation

```python

import re
import pandas as pd
from datasets import Dataset

DATASETS = {
"mt_bench_eval": r"https://raw.githubusercontent.com/kaistAI/prometheus/main/evaluation/benchmark/data/mt_bench_eval.json",
"vicuna_eval": r"https://raw.githubusercontent.com/kaistAI/prometheus/main/evaluation/benchmark/data/vicuna_eval.json",
"flask_eval": r"https://raw.githubusercontent.com/kaistAI/prometheus/main/evaluation/benchmark/data/flask_eval.json",
"feedback_eval": r"https://raw.githubusercontent.com/kaistAI/prometheus/main/evaluation/benchmark/data/feedback_collection_ood_test.json"
}

def extract_instruction_response(example):
pat1 = r'###The instruction to evaluate:(.*?)###Response to evaluate:'
pat2 = r'###Response to evaluate:(.*?)###Reference Answer'
matches_1 = re.search(pat1, example["instruction"], re.DOTALL)
matches_2 = re.search(pat2, example["instruction"], re.DOTALL)
instruction = response = None
if matches_1:
instruction = matches_1.group(1).strip()
if matches_2:
response = matches_2.group(1).strip()
return {
"instruction": instruction,
"response": response
}

if __name__ == "__main__":
dfs = []
for ds_name, filename in DATASETS.items():
if ds_name in {"mt_bench_eval", "vicuna_eval"}:
reader = lambda filename: pd.read_json(filename, lines=True)
else:
reader = lambda filename: pd.read_json(filename, orient="records")
print(f"Reading dataset: {ds_name}")
df = reader(DATASETS[ds_name])
df["base_dataset"] = ds_name
dfs.append(df)

df = pd.concat(dfs)

# Remove incomplete columns, we need to recompute those ourselves attending to our prompt format and model
df = df.drop(columns=["gpt4_score", "gpt4_feedback", "human_score"])

ds = Dataset.from_pandas(df, preserve_index=False).map(extract_instruction_response)

# Remove duplicates
df = ds.to_pandas()
df = df.loc[(df["instruction"] + "\n" + df["response"]).drop_duplicates().index]
df = df.reset_index().drop(columns="index")

# Add a hash id using instruction + response
df["hashid"] = (df["instruction"] + df["response"]).apply(lambda x: hashlib.sha256(x.encode("utf-8")).hexdigest())

# Reorder the columns and back to a Dataset
col_order = ["idx", "hashid", "response_source", "category", "criteria", "base_dataset", "instruction", "response"]
ds = Dataset.from_pandas(df[col_order], preserve_index=False)

ds.push_to_hub("distilabel-internal-testing/prometheus-bench-critique", split="test")
```

### Example instance

Some fields aren't filled as they weren't originally contained.

```json
{
"idx": 81,
"hashid": "8bb606d0e023413a4574ff34dfb4f4bb696f6076b847a682aefb8d3f7d85ca83",
"response_source": "chatgpt",
"category": "writing",
"criteria": null,
"base_dataset": "mt_bench_eval",
"instruction": "Here is a previous dialogue between an assistant and an user. Generate a human preferable response by the assistant.\nUser: Compose an engaging travel blog post about a recent trip to Hawaii, highlighting cultural experiences and must-see attractions.\n\nAssistant:Title: Aloha Adventures: Exploring the Vibrant Spirit of Hawaii\n\nIntroduction:\nAloha! Step into a world of tropical paradise, where the rhythm of the islands sweeps you off your feet. Recently, I embarked on a journey to the enchanting archipelago of Hawaii, a destination renowned for its stunning natural beauty, rich culture, and warm hospitality. From immersing myself in the vibrant local traditions to exploring must-see attractions, join me as I share my unforgettable experiences in this captivating travelogue.\n\n1. Embrace the Aloha Spirit:\nHawaii's true charm lies in its people and their genuine warmth. As soon as I set foot on the islands, I was greeted with a heartfelt \"Aloha\" from smiling locals. To truly embrace the Aloha Spirit, I recommend participating in a traditional lei-making workshop. Learn the art of creating beautiful flower garlands while connecting with the local culture. The lei serves as a symbol of welcome and is a delightful memento to carry throughout your trip.\n\n2. Discover the Pearl Harbor Historic Sites:\nImmerse yourself in the poignant history of Hawaii by visiting the Pearl Harbor Historic Sites. Explore the USS Arizona Memorial, a moving tribute to the lives lost during the infamous attack on Pearl Harbor. Allow yourself to be transported back in time as you walk through the exhibition halls and hear the stories of bravery and resilience. This experience is a powerful reminder of the importance of peace and understanding.\n\n3. Delve into Hawaiian Cuisine:\nHawaii's culinary scene is a melting pot of flavors, combining indigenous traditions with influences from diverse immigrant communities. Indulge your taste buds in the local delicacies like poke, a refreshing dish of marinated raw fish, or kalua pig, tender and smoky slow-roasted pork. For a truly authentic experience, attend a traditional Hawaiian luau, complete with a sumptuous feast, mesmerizing hula performances, and the soulful sounds of live music.\n\n4. Explore the Volcanoes National Park:\nNo trip to Hawaii is complete without a visit to the Volcanoes National Park, a geological wonderland where the primal forces of creation and destruction are on full display. Witness the mesmerizing lava flows, hike through unique landscapes, and marvel at the active Kilauea volcano. Remember to take a guided tour for insights into the scientific and cultural significance of these natural wonders.\n\n5. Chase Waterfalls:\nHawaii's lush rainforests are home to some of the most stunning waterfalls in the world. Venture into the\n\nUser: Rewrite your previous response. Start every sentence with the letter A.\n\nAssistant:",
"response": "Aloha! Allow me to take you on an incredible journey to Hawaii, where adventure and cultural immersion await. As I explored this enchanting archipelago, I discovered awe-inspiring attractions and absorbed the vibrant spirit of the islands.\n\nArriving in Hawaii, I was immediately greeted with the warm embrace of the Aloha Spirit. Everywhere I went, locals welcomed me with genuine smiles and heartfelt \"Aloha\"s, making me feel instantly at home.\n\nAmidst my travels, I had the opportunity to participate in a traditional lei-making workshop. Creating these beautiful flower garlands not only connected me to the local culture but also served as a cherished memento of my time in Hawaii.\n\nAt the Pearl Harbor Historic Sites, I delved into the island's profound history. Visiting the USS Arizona Memorial was an immensely moving experience, as I paid tribute to the brave souls who lost their lives during the infamous attack on Pearl Harbor. It reminded me of the importance of preserving peace and understanding.\n\nA journey to Hawaii wouldn't be complete without savoring its unique culinary delights. From indulging in the fresh and flavorful poke to relishing the succulent kalua pig, I immersed myself in the island's diverse cuisine. Attending a traditional Hawaiian luau was the icing on the cake, as I savored a sumptuous feast while being mesmerized by captivating hula performances.\n\nAs an adventurer, I couldn't resist exploring the wonders of Volcanoes National Park. The breathtaking landscapes and active Kilauea volcano left me in awe of nature's power. Guided tours provided valuable insights into the scientific and cultural significance of these volcanic marvels.\n\nAmidst Hawaii's lush rainforests, I discovered an abundance of cascading waterfalls. The sight of these majestic falls filled me with a sense of tranquility and wonder. Exploring these natural wonders allowed me to truly appreciate the island's untouched beauty.\n\nIn conclusion, my trip to Hawaii was an amazing adventure filled with cultural immersion and must-see attractions. The Aloha Spirit, Pearl Harbor Historic Sites, delectable cuisine, Volcanoes National Park, and enchanting waterfalls all added to the charm of this tropical paradise. Hawaii truly is a destination that captures the essence of adventure, and I can't wait to return for more unforgettable experiences. Aloha!"
}
```