talon-community/PanDomain-V1

--- license: cc-by-4.0 task_categories: - question-answering - text-generation language: - en pretty_name: PanDomain --- PanDomain-V1 is a high-quality, fully English dataset designed for training generalist language models across all major domains. It serves as the foundational training corpus for the Talon model family, built to support broad capabilities in both reasoning and generation. Every model sees everything. # Dataset Map Using Nomic [Click here to view the map](https://atlas.nomic.ai/data/overmind/pandomain-v1/map/fb4cb181-f665-4726-b45f-fab0353dff6d#hOyM) **Warning:** This dataset is contaminated with unloaded instructions. Some rows contain just "-" in the instruction field, while other rows are fine. This issue occurred because the dataset was not loaded properly. We apologize for the mistake. If you'd still however like to use this dataset, please use the following Python code to filter out the affected rows: ``` # 1. Install the datasets library if you haven't already # pip install datasets from datasets import load_dataset import logging logging.basicConfig(level=logging.INFO) logger = logging.getLogger(__name__) DATASET_NAME = "talon-community/PanDomain-V1" COLUMN_TO_CHECK = "conversations" VALUE_TO_REMOVE = "-" try: logger.info(f"Loading dataset: {DATASET_NAME}...") dataset = load_dataset(DATASET_NAME, trust_remote_code=True) logger.info("Dataset loaded successfully.") print("\nOriginal dataset structure:") print(dataset) first_split_name = next(iter(dataset)) if COLUMN_TO_CHECK not in dataset[first_split_name].features: logger.error(f"Error: Column '{COLUMN_TO_CHECK}' not found in dataset features.") print(f"\nAvailable columns are: {list(dataset[first_split_name].features.keys())}") print(f"Please ensure the dataset '{DATASET_NAME}' has a '{COLUMN_TO_CHECK}' column.") exit() def filter_sharegpt_function(example): conversation_list = example[COLUMN_TO_CHECK] if not isinstance(conversation_list, list): return True for turn in conversation_list: if isinstance(turn, dict) and 'value' in turn: if turn['value'] == VALUE_TO_REMOVE: return False return True logger.info(f"Filtering dataset...") filtered_dataset = dataset.filter(filter_sharegpt_function) logger.info("Filtering complete.") print(f"\nFiltered dataset structure:") print(filtered_dataset) except Exception as e: logger.error(f"An error occurred: {e}", exc_info=True) print(f"\nAn error occurred: {e}") print("Please check the dataset name, column name, data format, and your internet connection.")许可证：CC BY 4.0 任务类别： - 问答（Question Answering） - 文本生成（Text Generation） 语言： - 英语（en） 数据集展示名称：PanDomain PanDomain-V1是一款高质量全英文数据集，专为跨全主流领域训练通用大语言模型（Large Language Model, LLM）而设计。它作为Talon模型系列的基础训练语料库，旨在支撑模型在推理与生成两大方向上的全面能力。 全量数据对所有模型开放。 # 基于Nomic的数据集可视化映射 [点击此处查看映射图](https://atlas.nomic.ai/data/overmind/pandomain-v1/map/fb4cb181-f665-4726-b45f-fab0353dff6d#hOyM) **警告：** 本数据集存在未正确加载的指令污染问题。部分数据行的指令字段仅包含“-”，其余数据行则无异常。该问题源于数据集加载流程未正确执行，我们为此次失误致歉。 若您仍希望使用该数据集，请使用以下Python代码过滤受影响的数据行： python # 1. 若尚未安装，请先安装datasets库 # pip install datasets from datasets import load_dataset import logging logging.basicConfig(level=logging.INFO) logger = logging.getLogger(__name__) DATASET_NAME = "talon-community/PanDomain-V1" COLUMN_TO_CHECK = "conversations" VALUE_TO_REMOVE = "-" try: logger.info(f"正在加载数据集：{DATASET_NAME}...") dataset = load_dataset(DATASET_NAME, trust_remote_code=True) logger.info("数据集加载成功。") print(f" 原始数据集结构：") print(dataset) first_split_name = next(iter(dataset)) if COLUMN_TO_CHECK not in dataset[first_split_name].features: logger.error(f"错误：未在数据集特征中找到列'{COLUMN_TO_CHECK}'。") print(f" 可用列包括：{list(dataset[first_split_name].features.keys())}") print(f"请确保数据集'{DATASET_NAME}'包含'{COLUMN_TO_CHECK}'列。") exit() def filter_sharegpt_function(example): conversation_list = example[COLUMN_TO_CHECK] if not isinstance(conversation_list, list): return True for turn in conversation_list: if isinstance(turn, dict) and 'value' in turn: if turn['value'] == VALUE_TO_REMOVE: return False return True logger.info(f"正在过滤数据集...") filtered_dataset = dataset.filter(filter_sharegpt_function) logger.info("过滤完成。") print(f" 过滤后的数据集结构：") print(filtered_dataset) except Exception as e: logger.error(f"发生错误：{e}", exc_info=True) print(f" 发生错误：{e}") print("请检查数据集名称、列名、数据格式以及您的网络连接状况。")