Subjective Test Dataset and Meta-data-based Models for 360° Streaming Video Quality

During the last years, the number of 360° videos available for streaming has rapidly increased, leading to the need for 360° streaming video quality assessment. In this paper, we report and publish results of three subjective 360° video quality tests, with conditions used to reflect real-world bitrates and resolutions including 4K, 6K and 8K, resulting in 64 stimuli each for the first two tests and 63 for the third. As playout device we used the HTC Vive for the first and HTC Vive Pro for the remaining two tests. Video-quality ratings were collected using the 5-point Absolute Category Rating scale. The 360° dataset provided with the paper contains the links of the used source videos, the raw subjective scores, video-related meta-data, head rotation data and Simulator Sickness Questionnaire results per stimulus and per subject to enable reproducibility of the provided results. Moreover, we use our dataset to compare the performance of state-of-the-art full-reference quality metrics such as VMAF, PSNR, SSIM, ADM2, WS-PSNR and WS-SSIM. Out of all metrics, VMAF was found to show the highest correlation with the subjective scores. Further, we evaluated a center-cropped version of VMAF ("VMAF-cc") that showed to provide a similar performance as the full VMAF. In addition to the dataset and the objective metric evaluation, we propose two new video-quality prediction models, a bitstream meta-data-based model and a hybrid no-reference model using bitrate, resolution and pixel information of the video as input. The new lightweight models provide similar performance as the full-reference models while enabling fast calculations.近年来，可用于流式传输的360°视频数量快速增长，由此催生了360°流媒体视频质量评估的迫切需求。本文报告并发布了三项主观360°视频质量测试的结果，测试场景匹配真实流媒体的码率与分辨率参数，涵盖4K、6K与8K规格；前两项测试各包含64个测试素材，第三项测试则包含63个测试素材。测试所用的播放设备方面，第一项测试采用HTC Vive，后两项测试采用HTC Vive Pro。视频质量评分采用5级绝对类别评分（Absolute Category Rating）量表进行采集。本文配套的360°数据集包含所用源视频的链接、原始主观评分、视频相关元数据、受试者的头部旋转数据，以及每个测试素材对应的受试者模拟晕动症问卷（Simulator Sickness Questionnaire, SSQ）结果，以确保研究结果可复现。此外，本研究利用该数据集对比了VMAF、PSNR、SSIM、ADM2、WS-PSNR与WS-SSIM等当前主流全参考质量评价指标的性能表现。经测试，VMAF与主观评分的相关性在所有指标中最高。进一步地，本研究还评估了VMAF的中心裁剪变体（"VMAF-cc"），结果显示其性能与完整版VMAF相近。除数据集与客观指标评估外，本研究还提出了两款新型视频质量预测模型：一款基于比特流元数据的预测模型，以及一款以视频码率、分辨率与像素信息作为输入的混合无参考预测模型。这些新型轻量级模型不仅性能与全参考模型相近，还具备快速计算的优势。