Scicom-intl/Malaysian-Emilia-Dedup
收藏Hugging Face2026-01-25 更新2026-03-29 收录
下载链接：
https://hf-mirror.com/datasets/Scicom-intl/Malaysian-Emilia-Dedup
下载链接
链接失效反馈官方服务：
资源简介：
---
language:
- ms
- ta
- zh
---
# Malaysian-Emilia-Dedup
Global dedup of,
1. https://huggingface.co/datasets/mesolitica/Malaysian-Emilia-v2
2. https://huggingface.co/datasets/Scicom-intl/Malaysian-Chinese-Emilia
3. https://huggingface.co/datasets/Scicom-intl/Malaysian-Tamil-Emilia
Using FAISS for 80%, 85% and 90% similarities.
提供机构：
Scicom-intl
