SpeechPPL/SALMon_Llama-Mimi1.3B-normalized

--- configs: - config_name: bg_alignment data_files: - split: train path: bg_alignment/train-* - config_name: bg_all_consistency data_files: - split: train path: bg_all_consistency/train-* - config_name: bg_domain_consistency data_files: - split: train path: bg_domain_consistency/train-* - config_name: gender_consistency data_files: - split: train path: gender_consistency/train-* - config_name: rir_consistency data_files: - split: train path: rir_consistency/train-* - config_name: sentiment_alignment data_files: - split: train path: sentiment_alignment/train-* - config_name: sentiment_consistency data_files: - split: train path: sentiment_consistency/train-* - config_name: speaker_consistency data_files: - split: train path: speaker_consistency/train-* dataset_info: - config_name: bg_alignment features: - name: task dtype: string - name: ind dtype: int64 - name: positive_audio dtype: audio - name: negative_audio dtype: audio - name: prompt_audio dtype: audio: sampling_rate: 16000 - name: continuation_audio_positive dtype: audio: sampling_rate: 16000 - name: continuation_audio_negative dtype: audio: sampling_rate: 16000 - name: negative_audio_sanity dtype: audio: sampling_rate: 16000 - name: positive_sample_tokenwise_loss sequence: float32 - name: negative_sample_tokenwise_loss sequence: float32 - name: code_frame_rate dtype: int64 - name: code_depth dtype: int64 - name: model_sampling_rate sequence: int64 - name: ppl_sanity dtype: int64 - name: model_generated_continuation dtype: audio: sampling_rate: 24000 splits: - name: train num_bytes: 86750711 num_examples: 200 download_size: 86750711 dataset_size: 86750711 - config_name: bg_all_consistency features: - name: task dtype: string - name: ind dtype: int64 - name: positive_audio dtype: audio - name: negative_audio dtype: audio - name: audio_transition_s dtype: int64 - name: prompt_audio dtype: audio: sampling_rate: 16000 - name: continuation_audio_positive dtype: audio: sampling_rate: 16000 - name: continuation_audio_negative dtype: audio: sampling_rate: 16000 - name: negative_audio_sanity dtype: audio: sampling_rate: 16000 - name: positive_sample_tokenwise_loss sequence: float32 - name: negative_sample_tokenwise_loss sequence: float32 - name: positive_continuation_tokenwise_loss sequence: float32 - name: negative_continuation_tokenwise_loss sequence: float32 - name: prompt_sample_tokenwise_loss sequence: float32 - name: model_generated_continuation dtype: audio: sampling_rate: 24000 - name: code_frame_rate dtype: int64 - name: code_depth dtype: int64 - name: model_sampling_rate dtype: int64 - name: ppl_sanity dtype: int64 splits: - name: train num_bytes: 234883788 num_examples: 200 download_size: 234883788 dataset_size: 234883788 - config_name: bg_domain_consistency features: - name: task dtype: string - name: ind dtype: int64 - name: positive_audio dtype: audio - name: negative_audio dtype: audio - name: audio_transition_s dtype: int64 - name: prompt_audio dtype: audio: sampling_rate: 16000 - name: continuation_audio_positive dtype: audio: sampling_rate: 16000 - name: continuation_audio_negative dtype: audio: sampling_rate: 16000 - name: negative_audio_sanity dtype: audio: sampling_rate: 16000 - name: positive_sample_tokenwise_loss sequence: float32 - name: negative_sample_tokenwise_loss sequence: float32 - name: positive_continuation_tokenwise_loss sequence: float32 - name: negative_continuation_tokenwise_loss sequence: float32 - name: prompt_sample_tokenwise_loss sequence: float32 - name: model_generated_continuation dtype: audio: sampling_rate: 24000 - name: code_frame_rate dtype: int64 - name: code_depth dtype: int64 - name: model_sampling_rate dtype: int64 - name: ppl_sanity dtype: int64 splits: - name: train num_bytes: 237747052 num_examples: 200 download_size: 237747052 dataset_size: 237747052 - config_name: gender_consistency features: - name: task dtype: string - name: ind dtype: int64 - name: positive_audio dtype: audio - name: negative_audio dtype: audio - name: audio_transition_s dtype: int64 - name: prompt_audio dtype: audio: sampling_rate: 16000 - name: continuation_audio_positive dtype: audio: sampling_rate: 16000 - name: continuation_audio_negative dtype: audio: sampling_rate: 16000 - name: negative_audio_sanity dtype: audio: sampling_rate: 16000 - name: positive_sample_tokenwise_loss sequence: float32 - name: negative_sample_tokenwise_loss sequence: float32 - name: positive_continuation_tokenwise_loss sequence: float32 - name: negative_continuation_tokenwise_loss sequence: float32 - name: prompt_sample_tokenwise_loss sequence: float32 - name: model_generated_continuation dtype: audio: sampling_rate: 24000 - name: code_frame_rate dtype: int64 - name: code_depth dtype: int64 - name: model_sampling_rate dtype: int64 - name: ppl_sanity dtype: int64 splits: - name: train num_bytes: 238663168 num_examples: 200 download_size: 238663168 dataset_size: 238663168 - config_name: rir_consistency features: - name: task dtype: string - name: ind dtype: int64 - name: positive_audio dtype: audio - name: negative_audio dtype: audio - name: audio_transition_s dtype: int64 - name: prompt_audio dtype: audio: sampling_rate: 16000 - name: continuation_audio_positive dtype: audio: sampling_rate: 16000 - name: continuation_audio_negative dtype: audio: sampling_rate: 16000 - name: negative_audio_sanity dtype: audio: sampling_rate: 16000 - name: positive_sample_tokenwise_loss sequence: float32 - name: negative_sample_tokenwise_loss sequence: float32 - name: positive_continuation_tokenwise_loss sequence: float32 - name: negative_continuation_tokenwise_loss sequence: float32 - name: prompt_sample_tokenwise_loss sequence: float32 - name: model_generated_continuation dtype: audio: sampling_rate: 24000 - name: code_frame_rate dtype: int64 - name: code_depth dtype: int64 - name: model_sampling_rate dtype: int64 - name: ppl_sanity dtype: int64 splits: - name: train num_bytes: 218836796 num_examples: 200 download_size: 218836796 dataset_size: 218836796 - config_name: sentiment_alignment features: - name: task dtype: string - name: ind dtype: int64 - name: positive_audio dtype: audio - name: negative_audio dtype: audio - name: prompt_audio dtype: audio: sampling_rate: 16000 - name: continuation_audio_positive dtype: audio: sampling_rate: 16000 - name: continuation_audio_negative dtype: audio: sampling_rate: 16000 - name: negative_audio_sanity dtype: audio: sampling_rate: 16000 - name: positive_sample_tokenwise_loss sequence: float32 - name: negative_sample_tokenwise_loss sequence: float32 - name: code_frame_rate dtype: int64 - name: code_depth dtype: int64 - name: model_sampling_rate sequence: int64 - name: ppl_sanity dtype: int64 - name: model_generated_continuation dtype: audio: sampling_rate: 24000 splits: - name: train num_bytes: 46529917 num_examples: 200 download_size: 46529917 dataset_size: 46529917 - config_name: sentiment_consistency features: - name: task dtype: string - name: ind dtype: int64 - name: positive_audio dtype: audio - name: negative_audio dtype: audio - name: audio_transition_s dtype: int64 - name: prompt_audio dtype: audio: sampling_rate: 16000 - name: continuation_audio_positive dtype: audio: sampling_rate: 16000 - name: continuation_audio_negative dtype: audio: sampling_rate: 16000 - name: negative_audio_sanity dtype: audio: sampling_rate: 16000 - name: positive_sample_tokenwise_loss sequence: float32 - name: negative_sample_tokenwise_loss sequence: float32 - name: positive_continuation_tokenwise_loss sequence: float32 - name: negative_continuation_tokenwise_loss sequence: float32 - name: prompt_sample_tokenwise_loss sequence: float32 - name: model_generated_continuation dtype: audio: sampling_rate: 24000 - name: code_frame_rate dtype: int64 - name: code_depth dtype: int64 - name: model_sampling_rate dtype: int64 - name: ppl_sanity dtype: int64 splits: - name: train num_bytes: 232197295 num_examples: 200 download_size: 232197295 dataset_size: 232197295 - config_name: speaker_consistency features: - name: task dtype: string - name: ind dtype: int64 - name: positive_audio dtype: audio - name: negative_audio dtype: audio - name: audio_transition_s dtype: int64 - name: prompt_audio dtype: audio: sampling_rate: 16000 - name: continuation_audio_positive dtype: audio: sampling_rate: 16000 - name: continuation_audio_negative dtype: audio: sampling_rate: 16000 - name: negative_audio_sanity dtype: audio: sampling_rate: 16000 - name: positive_sample_tokenwise_loss sequence: float32 - name: negative_sample_tokenwise_loss sequence: float32 - name: positive_continuation_tokenwise_loss sequence: float32 - name: negative_continuation_tokenwise_loss sequence: float32 - name: prompt_sample_tokenwise_loss sequence: float32 - name: model_generated_continuation dtype: audio: sampling_rate: 24000 - name: code_frame_rate dtype: int64 - name: code_depth dtype: int64 - name: model_sampling_rate dtype: int64 - name: ppl_sanity dtype: int64 splits: - name: train num_bytes: 239774488 num_examples: 200 download_size: 239774488 dataset_size: 239774488 --- # SALMon Normalized Dataset This repo preserves the SALMon per-config folder layout while normalizing mismatched schema details across model families.

搜集汇总

数据集介绍

构建方式

SALMon_Llama-Mimi1.3B-normalized数据集是在原始SALMon数据集基础上，针对不同模型家族间存在的模式差异进行规范化处理而构建的。该数据集保留了原始SALMon的按配置划分的文件夹结构，涵盖八个核心配置子集：bg_alignment、bg_all_consistency、bg_domain_consistency、gender_consistency、rir_consistency、sentiment_alignment、sentiment_consistency及speaker_consistency。每个配置均包含训练集，每个训练集包含200个样本，提供正负样本音频、提示音频、延续音频、分词级损失序列、模型生成的延续音频以及采样率等信息。数据集的构建旨在为语音模型的对齐与一致性评估提供标准化、可复现的基准数据。

特点

该数据集的核心特点在于其多维度、精细化的评估能力。它不仅涵盖了背景对齐、领域一致性、性别一致性、房间脉冲响应一致性、情感对齐与一致性以及说话人一致性等多种任务，还通过正负样本对的设计，支持对模型在特定属性上的偏好判断。每个样本均包含详细的元数据，如音频过渡时间、分词级损失序列以及模型生成的延续音频，使得研究者能够深入分析模型在不同条件下的表现。数据集的规范性处理确保了不同模型家族间的可比性，为语音模型的对齐研究提供了统一且丰富的评测资源。

使用方法

使用该数据集时，用户可通过HuggingFace Datasets库加载所需的配置子集，例如使用`load_dataset('SALMon_Llama-Mimi1.3B-normalized', 'bg_alignment', split='train')`来加载背景对齐任务的训练数据。每个样本包含多个音频字段及对应的损失序列，可用于训练偏好模型或评估现有模型的对齐程度。研究者可根据任务需求选择相应的配置，利用正负样本对比、延续损失分析等方式，量化模型在背景、情感、说话人等方面的表现。数据集的规范化设计使得其可直接应用于模型训练和评估流程，无需额外的预处理步骤。

背景与挑战

背景概述

随着神经音频编解码模型与语言模型深度融合，生成式语音系统在自然度与表现力层面取得显著突破。然而，这类模型在长序列生成中常面临声学属性漂移、语义一致性崩塌等固有问题，亟需系统化的评估与调控手段。SALMon_Llama-Mimi1.3B-normalized数据集由语音生成与对齐领域的研究团队构建，旨在为多维度语音一致性评估提供标准化基准。该数据集涵盖背景对齐、说话人一致性、情感一致性、性别一致性、房间脉冲响应一致性及领域一致性八大子任务，通过精心设计的正负样本对与模型生成的延续音频，系统评估生成语音在语义、声学与情境属性上的保真度。自发布以来，该数据集已成为检验语音语言模型声学对齐能力的重要工具，推动了可控语音生成与属性一致性研究的发展。

当前挑战

该数据集的核心挑战在于解决语音语言模型在长序列生成中面临的属性一致性崩塌问题。在领域层面，现有模型难以同时维持背景声学特征、说话人身份、情感基调与性别特征等多维属性的长期连贯性，导致生成语音出现感知冲突或语义断裂。构建过程中，挑战同样显著：研究人员需要为每个子任务精确构造语义等价但声学属性相反的对比样本，并跨多个采样率（16kHz与24kHz）保持音频质量与时间对齐。此外，不同模型系列（如Mimi、Llama）间的架构差异要求在保留原始文件夹布局的前提下进行特征归并，确保token级损失、自回归连续性等结构化字段的兼容，该归一化过程需兼顾数据完整性与评估范式的跨模型迁移能力。

常用场景

经典使用场景

在语音生成与音频理解领域，该数据集专为评估与提升生成式语音模型的细粒度感知一致性而设计。其经典使用场景涵盖背景噪声对齐、背景全一致性、领域一致性、性别一致性、混响一致性、情感对齐、情感一致性以及说话人一致性共八类精细化任务。每一子集均包含成对的正负样本音频，并配以逐词损失与困惑度指标，用以量化模型在延续生成过程中对给定声学属性的保持能力。研究者可基于这些结构化数据，系统性地剖析当前语音生成模型在控制性生成方面的薄弱环节，进而优化其听觉连贯性与语义忠实度。

解决学术问题

该数据集直面语音生成模型在可控性评估与诊断方面的关键学术困境。传统评估指标多聚焦于音频质量或识别性能，而难以揭示模型在维持特定声学属性（如背景环境、情感色彩、说话人身份）时的真实鲁棒性。SALMon_Llama-Mimi1.3B-normalized通过引入正负样本逐词损失对比与成对测试范式，为研究者提供了量化模型属性一致性和对齐能力的高分辨率工具。其意义在于填补了语音生成模型细粒度诊断基准的空白，推动了从粗糙的整体评估向精细化、可分解的鲁棒性分析演进。

衍生相关工作

基于该数据集的结构化设计与评估范式，已催生出若干方向的衍生研究工作。一方面，研究者以其为基准，探索了基于对比学习的声学属性表征增强方法，通过在正负样本对之间构建隐空间约束来提升模型的控制敏感性。另一方面，有工作借鉴其逐词损失分析思路，提出了细粒度的语音生成归因算法，用以定位模型在何种时间步长上丢失了目标属性。此外，该数据集的子集划分方式促进了多任务持续学习框架的构建，使得单一模型能在共享参数下同时维持多种声学一致性，推动了统一可控语音生成模型的发展。