prnshv/teleembed-bench-clean

--- language: - en pretty_name: TeleEmbed Benchmark (Clean) tags: - retrieval - telecommunications - benchmarking license: apache-2.0 size_categories: - 1K<n<10K --- # TeleEmbed Benchmark — **Clean** track **Standalone dataset:** this repository is everything you need for the **clean** QA splits. Under **`TeleEmbed-Clean/`** you get both **`benchmark_*.json`** and the **passage corpora** (`chunks/<512|1024|2048>/chunks.json`) for O-RAN, 3GPP, and srsRAN. You do **not** need the Main dataset to run evaluation. **Companion dataset:** the **Main** track (different benchmark JSON, same underlying passages) is published separately; link it here when the URL is set, e.g. `https://huggingface.co/datasets/<your_org>/<your_main_dataset>`. --- ## What you must specify: the embedding model Use [Sentence Transformers](https://www.sbert.net/) via **`--model`** (Hub id or local path). The reference script encodes corpus + queries with the same encoder, L2-normalizes, and computes MRR / Recall@K. Always record which `--model` you used. --- ## Layout (this repo) ``` TeleEmbed-Clean/ oran/chunks/<512|1024|2048>/chunks.json oran/benchmark_*.json 3gpp/chunks/... 3gpp/benchmark_*.json srsran/chunks/... srsran/benchmark_*.json scripts/ evaluate_retrieval.py paths.py requirements.txt .gitattributes ``` Clone root = the folder that contains `TeleEmbed-Clean/` and `scripts/`. Run eval with **`--track clean`**. --- ## Quick start (scoring) ```bash python -m venv .venv && source .venv/bin/activate pip install -U pip && pip install -r requirements.txt cd scripts python evaluate_retrieval.py --corpus oran --track clean --chunk-size 512 \ --model intfloat/e5-base-v2 ``` --- ## Hugging Face download ```bash git clone https://huggingface.co/datasets/<YOUR_USER>/<THIS_REPO> cd <THIS_REPO> ``` --- ## Citation Cite this dataset URL/DOI and the Main benchmark dataset if both are used.