Subjective ratings and objective metric predictions of generic and personalized own voice reconstruction systems

This upload contains subjective ratings, objective metric predictions, and audio files for the preprint "Subjective quality evaluation of personalized own voice reconstruction systems". Abstract Own voice pickup technology for hearable devices facilitates communication in noisy environments. Own voice reconstruction (OVR) systems enhance the quality and intelligibility of the recorded noisy own voice signals. Since disturbances affecting the recorded own voice signals depend on individual factors, personalized OVR systems have the potential to outperform generic OVR systems. In this paper, we propose personalizing OVR systems through data augmentation and fine-tuning, comparing them to their generic counterparts. We investigate the influence of personalization on speech quality assessed by objective metrics and conduct a subjective listening test to evaluate quality under various conditions. In addition, we assess the prediction accuracy of the objective metrics by comparing predicted quality with subjectively measured quality. Our findings suggest that personalized OVR provides benefits over generic OVR for some talkers only. Our results also indicate that performance comparisons between systems are not always accurately predicted by objective metrics. In particular, certain disturbances lead to a consistent overestimation of quality compared to actual subjective ratings. Quality predictions Quality predictions are stored in the folder quality_predictions/. Each CSV file has these columns: - rating_stimulus (processing condition)- sentence (which sentence index, ranging from 0 to 2)- base_trial_id - this is the noise type in the noisy microphone signals- SNR the signal-to-noise ratio in dB at the outer microphone (here, always 0)- predicted_score the score predicted by the metric.- talker the talker in the recorded microphone signals. good talker corresponds to the high predicted benefit case, bad talker to the low predicted benefit case. Some of the metrics have their own CSV file, others are put together.Metrics:- LEAP (leap_predictions.csv)- eMoBi-Q (emobiq_predictions.csv)- PEMO-Q PSM (pemoq_predictions.csv)- WV-MOS (wvmos_predictions.csv)- PESQ, ESTOI, DNSMOS (SIG, BAK, OVRL, P808), SCORE-Q (MOS, distance) -> (pesq_estoi_dnsmos_scoreq_predictions.csv) This CSV file has an extra column metric containing the information which metric the predicted_score in each row belongs to Subjective ratings The subjective ratings are stored in subjective_ratings_mushra.csv. The CSV file has these columns: - rating_stimulus (processing condition)- sentence (which sentence index, ranging from 0 to 2)- base_trial_id - this is the noise type in the noisy microphone signals- rating_score the subjective MUSHRA rating, on a scale from 0 to 100- talker the talker in the recorded microphone signals. good talker corresponds to the high predicted benefit case, bad talker to the low predicted benefit case.- ID the subject/participant identifier Audio files The unprocessed and processed stimuli / audio signals used in the listening experiments are stored in the folder audio_files.Each subfolder fold0 and fold6 contains signals from the talkers with index 0 (low predicted benefit case) and 6 (high predicted benefit case). In each of these, there are subfolders containing signals for each noise type (e.g., factory_diffuse_0dBSNR).The filenames for the stimuli are as follows:- clean_reference_0.wav are the clean speech signals at the outer microphone without noise that were used as reference for the MUSHRA-like test. Here, the _0 indicates the sentence index (0,1,2).- noisy_anchor refers to the noisy outer microphone signal (unprocessed)- noisy_inear refers to the noisy in-ear microphone signal (unprocessed)- output_mwf refers to signals processed by the MWF- output_cv_eben_nonindiv_sim_nonindiv_finetune refers to signals processed by the EBEN generator DNN (trained with generic data augmentation and generic fine-tuning)- output_cv_nonindiv_sim refers to signals processed by the FT-JNF DNN (trained with generic data augmentation)- output_cv_nonindiv_sim_nonindiv_finetune refers to signals processed by the FT-JNF DNN (trained with generic data augmentation and generic fine-tuning)- output_cv_nonindiv_sim_indiv_finetune refers to signals processed by the FT-JNF DNN (trained with generic data augmentation and personalized fine-tuning)- output_cv_indiv_sim_indiv_finetune refers to signals processed by the FT-JNF DNN (trained with personalized data augmentation and personalized fine-tuning) Listening examples (online) Selected examples can also be played directly in the browser here: https://m-ohlenbusch.github.io/subjective_ovr_personalized/本上传内容包含预印本《个性化自声重建系统的主观音质评估》的主观评分、客观指标预测结果与音频文件。 摘要 自声拾取技术可辅助可穿戴听觉设备在嘈杂环境中实现顺畅沟通。自声重建（Own Voice Reconstruction，简称OVR）系统能够提升录制的嘈杂自声信号的音质与可懂度。由于影响录制自声信号的干扰因素存在个体差异，个性化OVR系统有望优于通用OVR系统。本文提出通过数据增强与微调实现OVR系统的个性化，并将其与通用OVR系统进行对比。我们研究了个性化对客观指标评估语音质量的影响，并开展主观听音测试以评估不同工况下的音质表现。此外，我们通过将预测音质与主观测得的音质进行对比，评估了各客观指标的预测准确性。研究结果表明，仅针对部分发话者，个性化OVR系统的表现优于通用OVR系统；同时，客观指标并非总能准确预测不同系统间的性能差异，部分干扰场景下，客观指标会持续高估实际主观评分对应的音质水平。 音质预测结果 音质预测结果存储于quality_predictions/文件夹中。每个CSV文件包含以下列： - rating_stimulus：处理条件标识 - sentence：句子索引，取值范围为0至2 - base_trial_id：带噪麦克风信号对应的噪声类型 - SNR：外置麦克风处的信噪比，单位为dB，本数据集固定为0 - predicted_score：对应指标的预测得分 - talker：录制麦克风信号对应的发话者。其中，good talker对应高预测收益场景，bad talker对应低预测收益场景。 部分指标拥有专属的CSV文件，其余指标则合并存储。涉及的指标如下： - LEAP：存储于leap_predictions.csv - eMoBi-Q：存储于emobiq_predictions.csv - PEMO-Q PSM：存储于pemoq_predictions.csv - WV-MOS：存储于wvmos_predictions.csv - PESQ、ESTOI、DNSMOS（包含子指标SIG、BAK、OVRL、P808）以及SCORE-Q（包含子指标MOS、distance）：上述指标合并存储于pesq_estoi_dnsmos_scoreq_predictions.csv，该文件额外包含metric列，用于标识每一行预测得分对应的指标类型。 主观评分 主观评分数据存储于subjective_ratings_mushra.csv文件中。该CSV文件包含以下列： - rating_stimulus：处理条件标识 - sentence：句子索引，取值范围为0至2 - base_trial_id：带噪麦克风信号对应的噪声类型 - rating_score：主观多刺激隐藏参考音频质量评价法（MUSHRA）评分，评分范围为0至100 - talker：录制麦克风信号对应的发话者。其中，good talker对应高预测收益场景，bad talker对应低预测收益场景 - ID：受试者/参与者唯一标识符。 音频文件 听音实验中使用的未处理与处理后的刺激音频信号存储于audio_files文件夹中。fold0与fold6两个子文件夹分别对应发话者索引0（低预测收益场景）与6（高预测收益场景）的测试音频信号。每个子文件夹下还包含对应各噪声类型的子文件夹（例如factory_diffuse_0dBSNR）。 测试音频的命名规则如下： - clean_reference_0.wav：作为类MUSHRA测试参考的无噪外置麦克风语音信号，其中_0表示句子索引（0、1、2）。 - noisy_anchor：未处理的带噪外置麦克风信号 - noisy_inear：未处理的带噪入耳麦克风信号 - output_mwf：经MWF处理后的信号 - output_cv_eben_nonindiv_sim_nonindiv_finetune：经EBEN生成式深度神经网络（Deep Neural Network，简称DNN）处理后的信号（采用通用数据增强与通用微调方式训练） - output_cv_nonindiv_sim：经FT-JNF深度神经网络（DNN）处理后的信号（采用通用数据增强方式训练） - output_cv_nonindiv_sim_nonindiv_finetune：经FT-JNF深度神经网络（DNN）处理后的信号（采用通用数据增强与通用微调方式训练） - output_cv_nonindiv_sim_indiv_finetune：经FT-JNF深度神经网络（DNN）处理后的信号（采用通用数据增强与个性化微调方式训练） - output_cv_indiv_sim_indiv_finetune：经FT-JNF深度神经网络（DNN）处理后的信号（采用个性化数据增强与个性化微调方式训练） 在线听音示例 可通过以下链接直接在浏览器中播放精选示例： https://m-ohlenbusch.github.io/subjective_ovr_personalized/