prnshv/teleembed-bench-main
收藏Hugging Face2026-04-08 更新2026-04-12 收录
下载链接：
https://hf-mirror.com/datasets/prnshv/teleembed-bench-main
下载链接
链接失效反馈官方服务：
资源简介：
---
language:
- en
pretty_name: TeleEmbed Benchmark (Main)
tags:
- retrieval
- telecommunications
- benchmarking
license: apache-2.0
size_categories:
- 1K<n<10K
---
# TeleEmbed Benchmark — **Main** track
Dense **embedding retrieval** benchmark over telecommunications corpora (**O-RAN**, **3GPP**, **srsRAN**). This repo ships **passage corpora** (`chunks.json`) and **labeled queries** (`benchmark_*.json`). **There is no fixed score in the files:** you choose a **sentence embedding model**, encode passages and questions, run retrieval, then read off MRR / Recall@K for **that** model.
**Companion dataset:** the **Clean** track (different benchmark JSON, **full copy** of the same passage corpora under its own `TeleEmbed-Clean/` tree) is published as a separate standalone dataset. Link it from this card when the URL is set, e.g. `https://huggingface.co/datasets/<your_org>/<your_clean_dataset>`.
---
## What you must specify: the embedding model
1. Install deps (`requirements.txt` includes `sentence-transformers`).
2. Run `scripts/evaluate_retrieval.py` with **`--model <id-or-path>`** — any [SentenceTransformer](https://www.sbert.net/)–compatible checkpoint (e.g. `intfloat/e5-base-v2`, `BAAI/bge-small-en-v1.5`, or a local folder).
3. The script encodes **every corpus chunk** and **every question** with that same model, L2-normalizes, ranks by dot product, and prints metrics.
4. **Publish or cite the exact `--model` string** (and corpus, `chunk_size`, `track`) next to your numbers. Omitting `--model` uses a tiny default only for smoke tests.
---
## Layout (this repo)
```
main/
oran/chunks/<512|1024|2048>/chunks.json
oran/benchmark_512.json (and 1024, 2048)
3gpp/...
srsran/...
scripts/
evaluate_retrieval.py
paths.py
requirements.txt
.gitattributes
```
---
## Task
For each sample: `question` = query embedding; corpus = all `chunk_text` in `chunks.json`; gold = `chunk_id`. Metrics: **MRR**, **Recall@K** (see script defaults for K).
---
## Quick start (scoring)
```bash
python -m venv .venv && source .venv/bin/activate
pip install -U pip && pip install -r requirements.txt
cd scripts
python evaluate_retrieval.py --corpus oran --track main --chunk-size 512 \
--model intfloat/e5-base-v2
```
Use `--device cuda`, `--batch-size`, `--top-k`, `--save-report`, `--max-samples` as needed.
---
## Hugging Face download
```bash
git clone https://huggingface.co/datasets/<YOUR_USER>/<THIS_REPO>
cd <THIS_REPO>
```
Use Git LFS (`git lfs install` before clone if needed).
---
## Citation
Cite this dataset URL/DOI once published, and the underlying specifications as appropriate.
提供机构：
prnshv
