O96a/opus-mt-arabic-benchmark-2026-03-28

---
license: apache-2.0
task_categories:
- translation
language:
- ar
- en
tags:
- translation
- arabic
- benchmark
- opus-mt
- nlp
pretty_name: OPUS-MT Arabic-English Translation Benchmark
size_categories:
- n<10
---

# OPUS-MT Arabic-English Translation Benchmark

## Experiment Details

- **Date:** 2026-03-28
- **Models Tested:**
- `Helsinki-NLP/opus-mt-en-ar` (English → Arabic)
- `Helsinki-NLP/opus-mt-ar-en` (Arabic → English)
- **Total Tests:** 9
- **Domain:** NLP / Translation

## Summary

| Metric | Value |
|--------|-------|
| MSA Accuracy Rate | 100% |
| Dialectal Accuracy Rate | 0% |
| Avg Latency (MSA) | 5.67s |
| Avg Latency (Dialectal) | 0.5s |

## Key Finding

**OPUS-MT handles Modern Standard Arabic (MSA) well but truncates Egyptian and Sudanese dialectal inputs.**

- Egyptian: "إزيك؟ كله تمام؟" → "I was gonna ask you something" (missed greeting entirely)
- Sudanese: "يا زول، كيف حالك؟ تعال نتغدا سوا" → "Hey, Zol, how are you?" (missed half)

## Results Table

| # | Direction | Type | Latency | Quality |
|---|-----------|------|---------|---------|
| 1 | EN→AR | Formal | 3.32s | ✅ Good |
| 2 | EN→AR | Technical | 13.48s | ✅ Good |
| 3 | EN→AR | Colloquial | 4.18s | ✅ Good |
| 4 | EN→AR | Code-switching | 8.97s | ✅ Good |
| 5 | AR→EN | MSA | 3.63s | ✅ Good |
| 6 | AR→EN | Technical | 3.75s | ✅ Good |
| 7 | AR→EN | Politeness | 3.88s | ✅ Good |
| 8 | AR→EN | **Egyptian dialect** | 0.42s | ❌ **Truncated** |
| 9 | AR→EN | **Sudanese dialect** | 0.58s | ❌ **Missed half** |

## Recommendations

1. Use OPUS-MT for MSA content only
2. Implement dialect detection before translation
3. Consider NLLB-200 or specialized dialect models for Arabic dialects
4. Add preprocessing for Egyptian, Sudanese, and other dialectal inputs

## References

- Model Discussion: [Helsinki-NLP/opus-mt-ar-en/discussions/10](https://huggingface.co/Helsinki-NLP/opus-mt-ar-en/discussions/10)
- Benchmark by: [O96a](https://huggingface.co/O96a)