vpermilp/nllb-200-1.3B-rust

---
language:
- ace
- acm
- acq
- aeb
- af
- ajp
- ak
- als
- am
- apc
- ar
- ars
- ary
- arz
- as
- ast
- awa
- ayr
- azb
- azj
- ba
- bm
- ban
- be
- bem
- bn
- bho
- bjn
- bo
- bs
- bug
- bg
- ca
- ceb
- cs
- cjk
- ckb
- crh
- cy
- da
- de
- dik
- dyu
- dz
- el
- en
- eo
- et
- eu
- ee
- fo
- fj
- fi
- fon
- fr
- fur
- fuv
- gaz
- gd
- ga
- gl
- gn
- gu
- ht
- ha
- he
- hi
- hne
- hr
- hu
- hy
- ig
- ilo
- id
- is
- it
- jv
- ja
- kab
- kac
- kam
- kn
- ks
- ka
- kk
- kbp
- kea
- khk
- km
- ki
- rw
- ky
- kmb
- kmr
- knc
- kg
- ko
- lo
- lij
- li
- ln
- lt
- lmo
- ltg
- lb
- lua
- lg
- luo
- lus
- lvs
- mag
- mai
- ml
- mar
- min
- mk
- mt
- mni
- mos
- mi
- my
- nl
- nn
- nb
- npi
- nso
- nus
- ny
- oc
- ory
- pag
- pa
- pap
- pbt
- pes
- plt
- pl
- pt
- prs
- quy
- ro
- rn
- ru
- sg
- sa
- sat
- scn
- shn
- si
- sk
- sl
- sm
- sn
- sd
- so
- st
- es
- sc
- sr
- ss
- su
- sv
- swh
- szl
- ta
- taq
- tt
- te
- tg
- tl
- th
- ti
- tpi
- tn
- ts
- tk
- tum
- tr
- tw
- tzm
- ug
- uk
- umb
- ur
- uzn
- vec
- vi
- war
- wo
- xh
- ydd
- yo
- yue
- zh
- zsm
- zu

language_details: "ace_Arab, ace_Latn, acm_Arab, acq_Arab, aeb_Arab, afr_Latn, ajp_Arab, aka_Latn, amh_Ethi, apc_Arab, arb_Arab, ars_Arab, ary_Arab, arz_Arab, asm_Beng, ast_Latn, awa_Deva, ayr_Latn, azb_Arab, azj_Latn, bak_Cyrl, bam_Latn, ban_Latn,bel_Cyrl, bem_Latn, ben_Beng, bho_Deva, bjn_Arab, bjn_Latn, bod_Tibt, bos_Latn, bug_Latn, bul_Cyrl, cat_Latn, ceb_Latn, ces_Latn, cjk_Latn, ckb_Arab, crh_Latn, cym_Latn, dan_Latn, deu_Latn, dik_Latn, dyu_Latn, dzo_Tibt, ell_Grek, eng_Latn, epo_Latn, est_Latn, eus_Latn, ewe_Latn, fao_Latn, pes_Arab, fij_Latn, fin_Latn, fon_Latn, fra_Latn, fur_Latn, fuv_Latn, gla_Latn, gle_Latn, glg_Latn, grn_Latn, guj_Gujr, hat_Latn, hau_Latn, heb_Hebr, hin_Deva, hne_Deva, hrv_Latn, hun_Latn, hye_Armn, ibo_Latn, ilo_Latn, ind_Latn, isl_Latn, ita_Latn, jav_Latn, jpn_Jpan, kab_Latn, kac_Latn, kam_Latn, kan_Knda, kas_Arab, kas_Deva, kat_Geor, knc_Arab, knc_Latn, kaz_Cyrl, kbp_Latn, kea_Latn, khm_Khmr, kik_Latn, kin_Latn, kir_Cyrl, kmb_Latn, kon_Latn, kor_Hang, kmr_Latn, lao_Laoo, lvs_Latn, lij_Latn, lim_Latn, lin_Latn, lit_Latn, lmo_Latn, ltg_Latn, ltz_Latn, lua_Latn, lug_Latn, luo_Latn, lus_Latn, mag_Deva, mai_Deva, mal_Mlym, mar_Deva, min_Latn, mkd_Cyrl, plt_Latn, mlt_Latn, mni_Beng, khk_Cyrl, mos_Latn, mri_Latn, zsm_Latn, mya_Mymr, nld_Latn, nno_Latn, nob_Latn, npi_Deva, nso_Latn, nus_Latn, nya_Latn, oci_Latn, gaz_Latn, ory_Orya, pag_Latn, pan_Guru, pap_Latn, pol_Latn, por_Latn, prs_Arab, pbt_Arab, quy_Latn, ron_Latn, run_Latn, rus_Cyrl, sag_Latn, san_Deva, sat_Beng, scn_Latn, shn_Mymr, sin_Sinh, slk_Latn, slv_Latn, smo_Latn, sna_Latn, snd_Arab, som_Latn, sot_Latn, spa_Latn, als_Latn, srd_Latn, srp_Cyrl, ssw_Latn, sun_Latn, swe_Latn, swh_Latn, szl_Latn, tam_Taml, tat_Cyrl, tel_Telu, tgk_Cyrl, tgl_Latn, tha_Thai, tir_Ethi, taq_Latn, taq_Tfng, tpi_Latn, tsn_Latn, tso_Latn, tuk_Latn, tum_Latn, tur_Latn, twi_Latn, tzm_Tfng, uig_Arab, ukr_Cyrl, umb_Latn, urd_Arab, uzn_Latn, vec_Latn, vie_Latn, war_Latn, wol_Latn, xho_Latn, ydd_Hebr, yor_Latn, yue_Hant, zho_Hans, zho_Hant, zul_Latn"

tags:
- nllb
- translation
license: "cc-by-nc-4.0"
datasets:
- flores-200
metrics:
- bleu
- spbleu
- chrf++
inference: false
---

# NLLB-200

This is the model card of NLLB-200's 1.3B variant.

Here are the [metrics](https://tinyurl.com/nllb200dense1bmetrics) for that particular checkpoint.

- Information about training algorithms, parameters, fairness constraints or other applied approaches, and features. The exact training algorithm, data and the strategies to handle data imbalances for high and low resource languages that were used to train NLLB-200 is described in the paper.
- Paper or other resource for more information NLLB Team et al, No Language Left Behind: Scaling Human-Centered Machine Translation, Arxiv, 2022
- License: CC-BY-NC
- Where to send questions or comments about the model: https://github.com/facebookresearch/fairseq/issues

## Intended Use
- Primary intended uses: NLLB-200 is a machine translation model primarily intended for research in machine translation, - especially for low-resource languages. It allows for single sentence translation among 200 languages. Information on how to - use the model can be found in Fairseq code repository along with the training code and references to evaluation and training data.
- Primary intended users: Primary users are researchers and machine translation research community.
- Out-of-scope use cases: NLLB-200 is a research model and is not released for production deployment. NLLB-200 is trained on general domain text data and is not intended to be used with domain specific texts, such as medical domain or legal domain. The model is not intended to be used for document translation. The model was trained with input lengths not exceeding 512 tokens, therefore translating longer sequences might result in quality degradation. NLLB-200 translations can not be used as certified translations.

## Metrics
• Model performance measures: NLLB-200 model was evaluated using BLEU, spBLEU, and chrF++ metrics widely adopted by machine translation community. Additionally, we performed human evaluation with the XSTS protocol and measured the toxicity of the generated translations.

## Evaluation Data
- Datasets: Flores-200 dataset is described in Section 4
- Motivation: We used Flores-200 as it provides full evaluation coverage of the languages in NLLB-200
- Preprocessing: Sentence-split raw text data was preprocessed using SentencePiece. The
SentencePiece model is released along with NLLB-200.

## Training Data
• We used parallel multilingual data from a variety of sources to train the model. We provide detailed report on data selection and construction process in Section 5 in the paper. We also used monolingual data constructed from Common Crawl. We provide more details in Section 5.2.

## Ethical Considerations
• In this work, we took a reflexive approach in technological development to ensure that we prioritize human users and minimize risks that could be transferred to them. While we reflect on our ethical considerations throughout the article, here are some additional points to highlight. For one, many languages chosen for this study are low-resource languages, with a heavy emphasis on African languages. While quality translation could improve education and information access in many in these communities, such an access could also make groups with lower levels of digital literacy more vulnerable to misinformation or online scams. The latter scenarios could arise if bad actors misappropriate our work for nefarious activities, which we conceive as an example of unintended use. Regarding data acquisition, the training data used for model development were mined from various publicly available sources on the web. Although we invested heavily in data cleaning, personally identifiable information may not be entirely eliminated. Finally, although we did our best to optimize for translation quality, mistranslations produced by the model could remain. Although the odds are low, this could have adverse impact on those who rely on these translations to make important decisions (particularly when related to health and safety).

## Caveats and Recommendations
• Our model has been tested on the Wikimedia domain with limited investigation on other domains supported in NLLB-MD. In addition, the supported languages may have variations that our model is not capturing. Users should make appropriate assessments.

## Carbon Footprint Details
• The carbon dioxide (CO2e) estimate is reported in Section 8.8.

原始信息汇总

数据集概述

数据集名称

• NLLB-200

支持的语言

• 200种语言，包括但不限于：ace, acm, acq, aeb, af, ajp, aka, amh, apc, arb, ars, ary, arz, asm, ast, awa, ayr, azb, azj, bak, bam, ban, bel, bem, ben, bho, bjn, bod, bos, bug, bul, cat, ceb, ces, cjk, ckb, crh, cym, dan, deu, dik, dyu, dzo, ell, eng, epo, est, eus, ewe, fao, pes, fij, fin, fon, fra, fur, fuv, gla, gle, glg, grn, guj, hat, hau, heb, hin, hne, hrv, hun, hye, ibo, ilo, ind, isl, ita, jav, jpn, kab, kac, kam, kan, kas, kat, kaz, kbp, kea, khm, kik, kin, kir, kmb, kon, kor, kmr, lao, lvs, lij, lim, lin, lit, lmo, ltg, ltz, lua, lug, luo, lus, mag, mai, mal, mar, min, mkd, mlt, mni, mos, mri, mya, nld, nno, nob, npi, nso, nus, nya, oci, gaz, ory, pag, pan, pap, pol, por, prs, pbt, quy, ron, run, rus, sag, san, sat, scn, shn, sin, slk, slv, smo, sna, snd, som, sot, spa, als, srd, srp, ssw, sun, swe, swh, szl, tam, tat, tel, tgk, tgl, tha, tir, taq, tpi, tsn, tso, tuk, tum, tur, twi, tzm, uig, ukr, umb, urd, uzn, vec, vie, war, wol, xho, ydd, yor, yue, zho, zul.

语言详情

• 每种语言支持的书写系统（如ace_Arab, ace_Latn等）。

标签

• nllb
• translation

许可证

• CC-BY-NC-4.0

包含的数据集

• Flores-200

评估指标

• BLEU
• spBLEU
• chrf++

模型用途

• 主要用途：用于机器翻译研究，特别是低资源语言的翻译。
• 主要用户：研究人员和机器翻译研究社区。
• 非预期用途：不用于生产部署，不用于特定领域的文本翻译，如医疗或法律领域，不用于文档翻译，不用于超过512个token的输入长度翻译，不作为认证翻译使用。

评估数据

• 数据集：Flores-200
• 预处理：使用SentencePiece进行句子分割的原始文本数据预处理。

训练数据

• 使用多种来源的并行多语言数据进行训练。

伦理考虑

• 考虑了低资源语言社区可能面临的风险，如信息误传或在线诈骗。
• 训练数据来自公开可用资源，可能包含未完全消除的个人识别信息。
• 尽管努力优化翻译质量，但模型可能产生误译，可能对依赖翻译做重要决策的人产生不利影响。

注意事项

• 模型在Wikimedia领域进行了测试，但未全面评估其他支持的领域。
• 支持的语言可能存在模型未捕捉到的变体。

碳足迹详情

• 二氧化碳排放量估计在第8.8节报告。

搜集汇总

数据集介绍

构建方式

在机器翻译领域，低资源语言的翻译质量始终是研究难点。vpermilp/nllb-200-1.3B-rust 数据集基于Meta发布的NLLB-200模型构建，该模型采用Transformer架构，参数规模达13亿。其训练数据融合了来自多种来源的平行语料与单语数据，其中单语语料源自Common Crawl。数据预处理阶段使用SentencePiece模型进行句子分割与分词，并通过精心设计的采样策略平衡高低资源语言的数据分布。整个构建过程遵循公平性约束，具体算法细节及数据不平衡处理策略均记录于NLLB团队发表的论文中。

特点

该数据集的核心特点在于其空前的语言覆盖广度，支持200种语言之间的单向翻译，涵盖从高资源语言到极低资源语言的广泛谱系，尤其侧重非洲语言。评价指标采用BLEU、spBLEU和chrF++等机器翻译领域通用标准，并辅以XSTS协议进行人工评估及翻译毒性检测。数据集的许可证为CC-BY-NC，适用于非商业研究场景。需要注意的是，模型训练输入长度限制为512个token，且未针对医疗、法律等专业领域优化，因此翻译质量可能受限于特定领域或长文本场景。

使用方法

使用该数据集时，研究者需通过Fairseq代码仓库获取模型调用方法，并参考随附的训练代码及评估数据引用。模型主要用于单句翻译任务，不适用于文档级翻译或生产环境部署。翻译前需对原始文本进行SentencePiece预处理，并确保输入长度不超过512个token。评估数据采用Flores-200数据集，该数据集提供了完整的200种语言覆盖。用户应谨慎对待翻译结果，尤其涉及健康、安全等关键决策时，需进行人工复核以避免误译风险。

背景与挑战

背景概述

在全球化的数字时代，机器翻译技术成为跨越语言壁垒、促进信息流通的关键工具。然而，全球数千种语言中，绝大多数属于低资源语言，缺乏充足的平行语料库支持，传统神经机器翻译模型难以覆盖这些语言。为此，Meta AI研究团队于2022年发布了NLLB-200（No Language Left Behind）项目，旨在通过大规模多语言翻译模型实现200种语言之间的互译，尤其关注非洲、亚洲及美洲的弱势语言。该模型的1.3B变体由vpermilp团队适配至Rust生态，以提升推理效率与可部署性。NLLB-200基于Flores-200评估数据集，采用BLEU、spBLEU和chrF++等指标进行性能度量，其研究核心在于解决低资源语言翻译的稀缺性问题，并通过人机协同评估确保翻译质量与伦理安全。这一工作对机器翻译领域产生了深远影响，推动了多语言包容性技术的发展。

当前挑战

NLLB-200所面临的挑战涵盖领域问题与构建过程两个层面。在领域问题方面，低资源语言的平行语料极度匮乏，导致模型在训练时难以捕获语言特有的语法与语义结构，翻译质量不稳定；同时，200种语言间的数据分布极不均衡，高资源语言（如英语、法语）的语料丰富，而许多非洲语言（如丰语、卢奥语）仅有少量样本，容易引发过拟合或欠拟合问题。在构建过程中，团队需从Common Crawl等公开来源挖掘并清洗多语言数据，但个人可识别信息难以完全剔除，带来隐私风险；此外，模型输入长度限制为512个token，长文本翻译时质量显著下降，且缺乏对医学、法律等专业领域的适配能力。这些挑战要求研究者在数据平衡、模型鲁棒性与伦理合规之间寻求精细权衡。

常用场景

经典使用场景

在机器翻译研究领域，NLLB-200-1.3B模型最经典的使用场景是作为多语言翻译的基准模型，尤其聚焦于低资源语言的翻译任务。该模型支持200种语言之间的单句翻译，覆盖了从广泛使用的语言到濒危语言的广阔谱系，为跨语言信息传递提供了统一框架。研究者常利用该模型在Flores-200数据集上进行评估，通过BLEU、spBLEU和chrF++等指标衡量翻译质量，从而探索不同语言对之间的翻译性能差异。这一场景不仅验证了模型在多语言环境下的泛化能力，还推动了低资源语言翻译技术的突破，成为后续研究的重要参照。

解决学术问题

该数据集解决了机器翻译领域中低资源语言数据匮乏的核心学术问题。传统翻译模型多聚焦于高资源语言，导致全球数千种语言在自然语言处理研究中被边缘化。NLLB-200通过整合多源平行语料和单语数据，采用创新性的数据平衡策略，显著提升了低资源语言的翻译质量。其意义在于打破了语言资源不均的壁垒，为跨语言信息平等获取提供了技术基础。这一工作不仅推动了多语言翻译理论的演进，还催化了人机交互中文化包容性的研究，对计算语言学和社会语言学交叉领域产生了深远影响。

衍生相关工作

NLLB-200衍生了一系列经典研究工作，包括多语言模型微调策略的探索、翻译质量评估指标的改进以及跨语言知识迁移方法的发展。例如，后续工作基于该模型提出了语言族谱感知的训练技术，进一步提升了相近语言之间的翻译一致性；同时，研究团队利用其输出构建了低资源语言的语料库，推动了无监督翻译范式的创新。此外，该模型还催生了针对翻译毒性检测和公平性约束的伦理研究，为负责任的AI部署提供了实践指导。这些衍生工作共同构建了从模型设计到社会影响评估的完整研究链条。