talon-community/PanDomain-V1

---
license: cc-by-4.0
task_categories:
- question-answering
- text-generation
language:
- en
pretty_name: PanDomain
---
PanDomain-V1 is a high-quality, fully English dataset designed for training generalist language models across all major domains. It serves as the foundational training corpus for the Talon model family, built to support broad capabilities in both reasoning and generation.

Every model sees everything.

# Dataset Map Using Nomic

[Click here to view the map](https://atlas.nomic.ai/data/overmind/pandomain-v1/map/fb4cb181-f665-4726-b45f-fab0353dff6d#hOyM)

**Warning:** This dataset is contaminated with unloaded instructions. Some rows contain just "-" in the instruction field, while other rows are fine. This issue occurred because the dataset was not loaded properly. We apologize for the mistake.

If you'd still however like to use this dataset, please use the following Python code to filter out the affected rows:

```
# 1. Install the datasets library if you haven't already
# pip install datasets

from datasets import load_dataset
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

DATASET_NAME = "talon-community/PanDomain-V1"
COLUMN_TO_CHECK = "conversations"
VALUE_TO_REMOVE = "-"

try:
logger.info(f"Loading dataset: {DATASET_NAME}...")
dataset = load_dataset(DATASET_NAME, trust_remote_code=True)
logger.info("Dataset loaded successfully.")
print("\nOriginal dataset structure:")
print(dataset)

first_split_name = next(iter(dataset))
if COLUMN_TO_CHECK not in dataset[first_split_name].features:
logger.error(f"Error: Column '{COLUMN_TO_CHECK}' not found in dataset features.")
print(f"\nAvailable columns are: {list(dataset[first_split_name].features.keys())}")
print(f"Please ensure the dataset '{DATASET_NAME}' has a '{COLUMN_TO_CHECK}' column.")
exit()

def filter_sharegpt_function(example):
conversation_list = example[COLUMN_TO_CHECK]
if not isinstance(conversation_list, list):
return True

for turn in conversation_list:
if isinstance(turn, dict) and 'value' in turn:
if turn['value'] == VALUE_TO_REMOVE:
return False
return True

logger.info(f"Filtering dataset...")
filtered_dataset = dataset.filter(filter_sharegpt_function)
logger.info("Filtering complete.")

print(f"\nFiltered dataset structure:")
print(filtered_dataset)

except Exception as e:
logger.error(f"An error occurred: {e}", exc_info=True)
print(f"\nAn error occurred: {e}")
print("Please check the dataset name, column name, data format, and your internet connection.")