Spreadthesign-Ten（SP-10）

迄今为止，大多数研究都集中在双语手语翻译 (BSLT) 上。但是，此类模型在构建多语言手语翻译系统方面效率低下。为了解决这个问题，我们介绍了多语言手语翻译 (MSLT) 任务。它旨在用单一模式完成多种手语和口语之间的翻译。然后，我们提出了第一个MSLT模型MLSLT，该模型包含两种新颖的动态路由机制，用于控制不同语言之间的参数共享程度。层内语言特定的路由控制从令牌级通过层内的软门流经共享参数和语言特定参数的数据的比例，和层间语言特定的路由通过层之间的软门控制和学习不同语言在语言级别的数据流路径。为了评估MLSLT的性能，我们收集了第一个公开可用的多语言手语理解数据集Spreadthesign-Ten (SP-10)，其中包含多达100种语言对，例如CSL->en，GSG->zh。实验结果表明，在许多情况下，MLSLT的平均性能优于基线MSLT模型和多个BSLT模型的组合。此外，我们还探索了手语中的零镜头翻译，发现我们的模型可以在某些语言对上实现与监督BSLT模型相当的性能。

Most existing research has focused on bilingual sign language translation (BSLT). However, such models are inefficient for constructing multilingual sign language translation systems. To address this limitation, we introduce the Multilingual Sign Language Translation (MSLT) task, which aims to enable translations between multiple sign languages and spoken languages using a single unified framework. We then propose the first MSLT model, dubbed MLSLT, which incorporates two novel dynamic routing mechanisms to regulate the degree of parameter sharing across different languages. The intra-layer language-specific routing controls the proportion of data flowing through shared and language-specific parameters at the token level via soft gates within each layer. The inter-layer language-specific routing, meanwhile, controls and learns the data flow paths for different languages at the language level through soft gates across layers. To evaluate the performance of MLSLT, we compile the first publicly available multilingual sign language understanding dataset Spreadthesign-Ten (SP-10), which covers up to 100 language pairs such as CSL->en and GSG->zh. Experimental results demonstrate that in many cases, the average performance of MLSLT outperforms baseline MSLT models and the ensemble of multiple BSLT models. Additionally, we explore zero-shot translation for sign languages and find that our model can achieve performance comparable to supervised BSLT models on certain language pairs.

搜集汇总

背景与挑战

背景概述

Spreadthesign-Ten（SP-10）是一个公开的多语言手语理解数据集，包含100种语言对，支持多语言手语翻译任务，由华为诺亚方舟实验室和浙江大学于2022年发布。