Bagpiper_SFT_Data
收藏资源简介：
Bagpiper SFT Data 是 Bagpiper 模型的监督微调语料库，Bagpiper 是一个开放式的音频语言模型，能够通过丰富的文本描述和规划来理解和生成语音、音乐、环境声音及其混合物。该数据集包含两个配置：generation 配置将自然语言请求转换为规划、丰富描述和音频，用于开放式语音、音乐、声音和混合音频生成序列；understanding 配置结合音频和请求，输出丰富描述、推理和答案，用于音频问答、多项选择推理和转录序列。数据以 Parquet 格式存储，每行包含完整的文本对话和编码后的音频字节，无需外部下载。数据集总共有约 2,667,017 行，分为 6,804 个分片，Parquet 文件总大小约 1.13 TB，嵌入音频字节约 1.66 TB。generation 配置包含 1,474,011 行，来源包括 part2_gen_v1_realistic 等子集；understanding 配置包含 1,193,006 行，来源包括 airbench_train_v1 等子集。数据通过从丰富描述的音频合成构建，并使用大语言模型进行请求/推理模拟和质量过滤。已知局限包括：机器生成的描述可能存在幻觉或偏见；部分音频与 AIR-Bench 和 AudioBench 基准存在指纹重叠；音频来源具有异质性且许可不一；语料库可能包含合成或转换的音频，未经过完整的人工审核；未全面筛查个人信息、不安全内容或冒犯性语言。该数据集适用于音频理解与生成任务，尤其是开放式音频任务。
Bagpiper SFT Data is a supervised fine-tuning corpus for the Bagpiper model, an open-ended audio language model capable of understanding and generating speech, music, environmental sounds, and their mixtures through rich textual descriptions and planning. The dataset includes two configurations: the generation configuration converts natural language requests into planning, rich descriptions, and audio for open-ended speech, music, sound, and mixed audio generation sequences; the understanding configuration combines audio and requests to output rich descriptions, reasoning, and answers for audio question answering, multiple-choice reasoning, and transcription sequences. Data is stored in Parquet format, with each row containing complete text dialogue and encoded audio bytes, requiring no external downloads. The dataset contains approximately 2,667,017 rows across 6,804 shards, with Parquet files totaling about 1.13 TB and embedded audio bytes about 1.66 TB. The generation configuration has 1,474,011 rows, sourced from subsets such as part2_gen_v1_realistic; the understanding configuration has 1,193,006 rows, sourced from subsets such as airbench_train_v1. Data is constructed from audio synthesis with rich descriptions and uses large language models for request/reasoning simulation and quality filtering. Known limitations include: machine-generated descriptions may contain hallucinations or biases; some audio overlaps with AIR-Bench and AudioBench benchmarks; audio sources are heterogeneous with varying licenses; the corpus may contain synthetic or converted audio without full human review; personal information, unsafe content, or offensive language have not been fully screened. This dataset is suitable for audio understanding and generation tasks, especially open-ended audio tasks.
Bagpiper SFT 数据集中文概述
数据集简介
Bagpiper SFT Data 是 Bagpiper（一个开放音频语言模型，能够通过丰富的文本描述理解和生成语音、音乐、环境声音及其混合内容）的监督微调（SFT）语料库。
数据集配置
数据集公开版本包含两种配置：
| 配置 | 方向 | 内容 |
|---|---|---|
generation |
自然语言请求 → 规划、丰富描述、音频 | 开放式语音、音乐、声音及混合音频的生成序列 |
understanding |
音频和请求 → 丰富描述、推理、答案 | 音频问答、多选推理和转录序列 |
数据规模
| 分区 | 行数 | 分片数 | Parquet 字节数 | 嵌入音频字节数 |
|---|---|---|---|---|
generation |
1,474,011 | 1,944 | 363,645,830,327 | 411,563,826,060 |
understanding |
1,193,006 | 4,860 | 768,716,849,416 | 1,251,578,476,222 |
| 总计 | 2,667,017 | 6,804 | 1,132,362,679,743 | 1,663,142,302,282 |
- 数据集总大小约为 1.13 TB（Parquet 字节），嵌入音频字节总计约 1.66 TB
- 每行数据自包含：有序文本对话和完整编码的音频字节存储在同一 Parquet 行中
- 数据被分割为多个确定性的 Parquet 分片（每片约 256 MiB），无需外部媒体下载或表连接
数据来源分布
| 分区 | 来源子集 | 行数 |
|---|---|---|
| generation | part2_gen_v1_realistic |
463,850 |
| generation | part2_gen_v1_imaginary |
371,827 |
| generation | part3_gen_v1_realistic |
95,134 |
| generation | part3_gen_v1_imaginary |
72,508 |
| generation | part4_gen_v1_realistic |
268,584 |
| generation | part4_gen_v1_imaginary |
202,108 |
| understanding | airbench_train_v1 |
357,896 |
| understanding | mmau_train_v1 |
334,224 |
| understanding | asr_v2_inverse_200k |
200,000 |
| understanding | audiobench_train_v1 |
300,886 |
数据构建方式
- 数据示例由带丰富描述的音频合成生成
- 理解序列：结合输入音频片段和任务请求，以及丰富描述、推理轨迹和答案
- 生成序列：将请求扩展为规划、丰富描述，再生成目标音频
- 项目使用大语言模型进行请求/推理模拟和质量过滤
数据加载方式
使用 Hugging Face datasets 库加载，需指定 Parquet 文件路径并建议使用流式加载：
python from datasets import load_dataset
generation = load_dataset( "parquet", data_files={"train": "hf://datasets/espnet/Bagpiper_SFT_Data/generation/.parquet"}, split="train", streaming=True, ) understanding = load_dataset( "parquet", data_files={"train": "hf://datasets/espnet/Bagpiper_SFT_Data/understanding/.parquet"}, split="train", streaming=True, )
已知局限
- 标题、推理和质量判断为机器生成，可能包含幻觉或偏见
- 论文报告 SFT 输入与 AudioBench（12.8%）和 AIR-Bench（3.2%）存在音频指纹重叠，相关基准结果需谨慎解读
- 源音频具有异构来源和许可，未确认再分发权利的配置或示例将被扣留
- 语料库可能包含合成或转换的音频，不应视为经过验证的人工标注
- 理解分区的 v1 混合规模大于论文中报告的约 84.5 万条理解序列，论文 v2 对齐已推迟
- 语料库未彻底审核个人信息、不安全内容、攻击性语言或转录中嵌入的受版权保护文本
相关资源
- 论文：Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions
- 项目主页：https://bagpiper-cmu.github.io/
- 基础模型：espnet/bagpiper
- 预训练数据：espnet/Bagpiper_PreTrain_Data
- ESPnet：espnet/espnet
