general-pmd-synthetic-testing

This dataset is designed to be used in testing. It's derived from general-pmd/localized_narratives__ADE20k dataset

The current splits are: `['100.unique', '100.repeat', '300.unique', '300.repeat', '1k.unique', '1k.repeat', '10k.unique', '10k.repeat']`.

The `unique` ones ensure uniqueness across `text` entries.

The `repeat` ones are repeating the same 10 unique records: - these are useful for memory leaks debugging as the records are always the same and thus remove the record variation from the equation.

The default split is `100.unique`

The full process of this dataset creation, including which records were used to build it, is documented inside [general-pmd-synthetic-testing.py](https://huggingface.co/datasets/HuggingFaceM4/general-pmd-synthetic-testing/blob/main/general-pmd-synthetic-testing.py)

本数据集专为测试场景打造，衍生自`general-pmd/localized_narratives__ADE20k`数据集。

当前预设的数据划分集合为：`['100.unique', '100.repeat', '300.unique', '300.repeat', '1k.unique', '1k.repeat', '10k.unique', '10k.repeat']`。

其中带`unique`后缀的划分可保证所有文本条目均具备唯一性。

带`repeat`后缀的划分则重复使用同一组10条唯一记录：此类划分适用于内存泄漏调试场景，因其记录始终保持一致，可排除记录差异对实验结果的干扰。

默认的数据划分方式为`100.unique`。

本数据集的完整构建流程（包含所采用的记录来源）已在[general-pmd-synthetic-testing.py](https://huggingface.co/datasets/HuggingFaceM4/general-pmd-synthetic-testing/blob/main/general-pmd-synthetic-testing.py)中进行了详细说明。

搜集汇总

背景与挑战

背景概述

该数据集专用于测试目的，源自general-pmd/localized_narratives__ADE20k数据集，包含多种分割类型，如unique和repeat，以支持不同测试场景，其中unique分割确保文本条目唯一性，repeat分割则用于内存泄漏调试。默认分割为100.unique，创建过程在相关Python文件中详细记录。