LIACC/Emakhuwa-FLORES

--- license: cc-by-sa-4.0 language: - pt - vmw --- # Dataset card ## Description <!-- A concise description of the data associated with this card. --> FLORES+ dev and devtest set in Emakhuwa ## License <!-- Contributions to existing datasets must be released under the same license as the parent dataset. For completely new contributions, we encourage the use of an open license. At a minimum, data should be made available for research use. Please specify the license using an SPDX license identifier. --> CC-BY-SA-4.0 ## Attribution <!-- Who should be credited for creating this dataset? Feel free to include citation data in BibTeX format. --> ```bibtex @inproceedings{ali-etal-2024-expanding, title = "Expanding {FLORES}+ Benchmark for More Low-Resource Settings: {P}ortuguese-Emakhuwa Machine Translation Evaluation", author = "Ali, Felermino Dario Mario and Lopes Cardoso, Henrique and Sousa-Silva, Rui", editor = "Haddow, Barry and Kocmi, Tom and Koehn, Philipp and Monz, Christof", booktitle = "Proceedings of the Ninth Conference on Machine Translation", month = nov, year = "2024", address = "Miami, Florida, USA", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2024.wmt-1.45", pages = "579--592", abstract = "As part of the Open Language Data Initiative shared tasks, we have expanded the FLORES+ evaluation set to include Emakhuwa, a low-resource language widely spoken in Mozambique. We translated the \textit{dev} and \textit{devtest} sets from Portuguese into Emakhuwa, and we detail the translation process and quality assurance measures used. Our methodology involved various quality checks, including post-editing and adequacy assessments. The resulting datasets consist of multiple reference sentences for each source. We present baseline results from training a Neural Machine Translation system and fine-tuning existing multilingual translation models. Our findings suggest that spelling inconsistencies remain a challenge in Emakhuwa. Additionally, the baseline models underperformed on this evaluation set, underscoring the necessity for further research to enhance machine translation quality for Emakhuwa.The data is publicly available at \url{https://huggingface.co/datasets/LIACC/Emakhuwa-FLORES}", } ``` ## Abstract As part of the Open Language Data Initiative shared tasks, we have expanded the FLORES+ evaluation set to include Emakhuwa, a low-resource language widely spoken in Mozambique. We translated the dev and devtest sets from Portuguese into Emakhuwa, and we detail the translation process and quality assurance measures used. Our methodology involved various quality checks, including post-editing and adequacy assessments. The resulting datasets consist of multiple reference sentences for each source. We present baseline results from training a Neural Machine Translation system and fine-tuning existing multilingual translation models. Our findings suggest that spelling inconsistencies remain a challenge in Emakhuwa. Additionally, the baseline models underperformed on this evaluation set, underscoring the necessity for further research to enhance machine translation quality for Emakhuwa. ## Language codes <!-- * If this language is assigned an ISO 639-3 individual language code (not a macrolanguage code), specify it here. * Please specify the script this language is written in using an ISO 15924 code. * If this language is assigned a Glottocode, please specify it here. --> - **Language(s) (NLP):** Emakhuwa (vmw), Portuguese (pt) * ISO 639-3: vmw * ISO 15924: Latn * Glottocode: cent2033 ## Additional language information <!-- Any relevant additional information on the language, such as: * A list of reference publications and software (dictionaries, grammars, spellcheckers). * If applicable, any additional information about dialectal variation that is not captured by the Glottocode. * If relevant, the orthography used in your contribution. --> ## Workflow <!-- What workflow was followed in creating this dataset? E.g., for a translated dataset, relevant information includes: what language the content was translated from, the number of translators, aggregate translator information (how many were native speakers in the target language, how many were highly proficient in the target languages, how many had professional translation experience), was any fraction of the data checked independently by third parties, etc. --> Data was translated from Portuguese by 2 translators, all bilingual speakers of the languages. All translators were professional translators. 100% of the data was checked by three more independent translator. The workflow is divided into three main steps: 1. **Data Preparation**: - Sentences from the *devtest* and *dev* sets are compiled into segments and loaded into the Matecat CAT tool. - Guidelines and a glossary were prepared to standardize the translation process. The guidelines were adapted from the OLDI guidelines and written in Portuguese, focusing on the central variant of Emakhuwa. The glossary was created by digitizing existing bilingual dictionaries and a glossary from Radio of Mozambique, ensuring consistent translations and minimizing loanword use. 2. **Translation**: - The translation tasks were divided between two translators. They used a spell checker system to identify potential misspellings, which were then corrected based on feedback. 3. **Validation**: - This step included revision and judgments. The translated works were exchanged between translators for post-editing. - Direct Assessment was also used, where three raters evaluated the translation's adequacy on a scale from 0 to 100, to measure how well the translations preserved the original meaning. ## Additional guidelines <!-- Were any additional guidelines agreed upon? Examples might include style guidelines, the use of particular grammatical forms or sentence structures, specific spelling or punctuation rules to be followed, etc. --> We also requested translators to mark loanwords that were adapted into Emakhuwa during the translation of each segment. - **Funded by:** This dataset was created with support from Lacuna Fund, the world’s first collaborative effort to provide data scientists, researchers, and social entrepreneurs in low- and middle-income contexts globally with the resources they need to produce labeled datasets that address urgent problems in their communities. Lacuna Fund is a funder collaborative that includes The Rockefeller Foundation, Google.org, Canada’s International Development Research Centre, the German Federal Ministry for Economic Cooperation and Development (BMZ) with GIZ as implementing agency, Wellcome Trust, Gordon and Betty Moore Foundation, Patrick J. McGovern Foundation, and The Robert Wood Johnson Foundation. See https://lacunafund.org/about/ for more information.许可证：CC BY-SA 4.0 语言： - 葡萄牙语（pt） - 埃马库瓦语（vmw） # 数据集卡片（Dataset Card） ## 数据说明 <!-- 对本卡片关联数据的简明描述。 --> 本数据集为埃马库瓦语（Emakhuwa）的FLORES+开发集（dev）与开发测试集（devtest）。 ## 许可证 <!-- 对现有数据集的贡献必须遵循与原数据集相同的许可证协议。对于全新贡献，我们鼓励采用开放许可证协议。数据至少应可用于研究用途。请使用SPDX许可证标识符指定许可证。 --> CC-BY-SA-4.0 ## 归属说明 <!-- 本数据集的创建者应如何标注？可包含BibTeX格式的引用数据。 --> bibtex @inproceedings{ali-etal-2024-expanding, title = "Expanding {FLORES}+ Benchmark for More Low-Resource Settings: {P}ortuguese-Emakhuwa Machine Translation Evaluation", author = "Ali, Felermino Dario Mario and Lopes Cardoso, Henrique and Sousa-Silva, Rui", editor = "Haddow, Barry and Kocmi, Tom and Koehn, Philipp and Monz, Christof", booktitle = "Proceedings of the Ninth Conference on Machine Translation", month = nov, year = "2024", address = "Miami, Florida, USA", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2024.wmt-1.45", pages = "579--592", abstract = "As part of the Open Language Data Initiative shared tasks, we have expanded the FLORES+ evaluation set to include Emakhuwa, a low-resource language widely spoken in Mozambique. We translated the extit{dev} and extit{devtest} sets from Portuguese into Emakhuwa, and we detail the translation process and quality assurance measures used. Our methodology involved various quality checks, including post-editing and adequacy assessments. The resulting datasets consist of multiple reference sentences for each source. We present baseline results from training a Neural Machine Translation system and fine-tuning existing multilingual translation models. Our findings suggest that spelling inconsistencies remain a challenge in Emakhuwa. Additionally, the baseline models underperformed on this evaluation set, underscoring the necessity for further research to enhance machine translation quality for Emakhuwa.The data is publicly available at url{https://huggingface.co/datasets/LIACC/Emakhuwa-FLORES}", } ## 摘要 作为开放语言数据倡议（Open Language Data Initiative）共享任务的一部分，我们对FLORES+评估集进行了扩展，新增了莫桑比克广泛使用的低资源语言埃马库瓦语。我们将开发集与开发测试集从葡萄牙语译为埃马库瓦语，并详细阐述了所用的翻译流程与质量保障措施。本研究采用了包括后编辑与充分性评估在内的多重质量检查手段。最终生成的数据集为每个源句提供了多条参考译文。 我们报告了基于神经机器翻译（Neural Machine Translation）系统训练与对现有多语言翻译模型进行微调得到的基准结果。研究结果表明，拼写不一致仍是埃马库瓦语翻译面临的挑战。此外，基准模型在本评估集上表现欠佳，凸显了针对埃马库瓦语开展进一步研究以提升机器翻译质量的必要性。本数据集已公开部署于：url{https://huggingface.co/datasets/LIACC/Emakhuwa-FLORES} ## 语言代码 <!-- * 若该语言已分配ISO 639-3独立语言代码（非宏语言代码），请在此处指定。 * 请使用ISO 15924代码指定该语言的书写脚本。 * 若该语言已分配Glottocode代码，请在此处指定。 --> - **自然语言处理所用语言：** 埃马库瓦语（vmw）、葡萄牙语（pt） * ISO 639-3代码：vmw * ISO 15924书写脚本代码：Latn（拉丁字母） * Glottocode代码：cent2033 ## 补充语言信息 <!-- 与该语言相关的任何其他信息，例如： * 参考出版物与软件列表（词典、语法书、拼写检查工具等）。 * 若适用，补充未在Glottocode中体现的方言变体相关信息。 * 若相关，说明本贡献中使用的正字法。 --> 无 ## 制作流程 <!-- 创建本数据集时遵循了何种流程？例如，对于翻译数据集，相关信息包括：内容从何种语言翻译而来、译者数量、译者的综合信息（有多少人为目标语母语使用者、有多少人精通目标语言、有多少人拥有专业翻译经验）、是否有部分数据由第三方独立审核等。 --> 本数据集由2名双语译者从葡萄牙语翻译而来，二人均为专业译者。全部数据均由另外3名独立译者完成审核。 本制作流程分为三个主要步骤： 1. **数据准备**： - 将开发测试集与开发集的句子拆分为分段并加载至Matecat计算机辅助翻译（Computer-Aided Translation, CAT）工具中。 - 制定翻译指南与术语表以统一翻译流程。本指南改编自开放语言数据倡议指南，以葡萄牙语撰写，聚焦埃马库瓦语的标准变体。术语表通过数字化现有双语词典与莫桑比克广播电台的术语表创建而成，旨在确保翻译一致性并减少外来词的使用。 2. **翻译执行**： - 翻译任务由两名译者分工完成。译者使用拼写检查系统识别潜在拼写错误，并基于反馈完成修正。 3. **验证审核**： - 本步骤包含修订与质量评判。译者间交换译稿以完成后编辑工作。 - 同时采用直接评估法：由3名评分者以0至100的量表评估译文的充分性，衡量译文对原文语义的保留程度。 ## 额外翻译指南 <!-- 是否商定了额外的翻译准则？例如风格指南、特定语法形式或句式的使用要求、需遵循的特定拼写或标点规则等。 --> 我们同时要求译者在翻译每个分段时，标注出被适配为埃马库瓦语的外来词。 - **资助方：** 本数据集的制作得到了拉库纳基金（Lacuna Fund）的支持。该基金是全球首个面向中低收入地区的数据科学家、研究人员与社会创业者的协作项目，旨在为其提供所需资源，以创建能够解决本地紧迫问题的标注数据集。Lacuna Fund是一个资助者联盟，成员包括洛克菲勒基金会、Google.org、加拿大国际发展研究中心、德国联邦经济合作与发展部（BMZ，由GIZ作为执行机构）、惠康基金会、戈登与贝蒂·摩尔基金会、帕特里克·J·麦戈文基金会以及罗伯特·伍德·约翰逊基金会。更多信息请参见：https://lacunafund.org/about/