Neural-Navigator-Labs/wmt14_en-de
收藏Hugging Face2026-03-20 更新2026-03-29 收录
下载链接：
https://hf-mirror.com/datasets/Neural-Navigator-Labs/wmt14_en-de
下载链接
链接失效反馈官方服务：
资源简介：
---
language:
- en
- de
task_categories:
- translation
pretty_name: SmolTrans EN-DE Clean Dataset
size_categories:
- 1M<n<10M
---
# SmolTrans EN-DE Dataset
High-quality English–German parallel dataset filtered using:
- Language detection
- HTML removal
- Duplicate removal
- LM perplexity filtering
- Perplexity ratio filtering
- Embedding cosine similarity
## Statistics
- Total pairs: 4.5M
- Median length: ...
- Filtering removed: ~X%
## Format
{"en": "...", "de": "..."}
## Usage
from datasets import load_dataset
ds = load_dataset("<your-username>/smoltrans-en-de")
## Notes
Short sentences may show unstable perplexity values.
提供机构：
Neural-Navigator-Labs
