SlideOn-sparse-vectors

# SlideOn sparse vectors for MS MARCO passage and BEIR Pre-computed learned sparse vectors from [lightonai/SlideOn](https://huggingface.co/lightonai/SlideOn) (revision `370c600`) for the MS MARCO v1 passage corpus and the 29 BEIR v1.0.0 corpora distributed with [Anserini](https://github.com/castorini/anserini), plus their evaluation queries. Document ids, query ids and texts follow Anserini exactly (BEIR documents are encoded as `title + " " + text`), so the vectors drop into Anserini/Pyserini impact indexes. See the [SlideOn & mSlideOn blog post](https://huggingface.co/blog/lightonai/slideon-mslideon) for the models. ## Layout One config per dataset; each row is one document or query: | column | type | content | | --- | --- | --- | | `id` | string | Anserini docid / qid | | `indices` | list<uint16> | active latents (0–32767), **sorted by decreasing weight** | | `weights` | list<float16> | the model's weights, exactly as it outputs them (FP16, no quantization) | Scores are raw dot products between query and document vectors (no log, no normalization). ## Choosing the sparsity (Matryoshka budgets) SlideOn was trained with Matryoshka budgets of 512 or 1024 document latents and 32, 64 or 128 query latents. Because each row is sorted by weight, keeping the first `k` entries is exactly the model's `max_active_dims=k` output: - **documents** store the top **1024** latents (the largest trained document budget; docs with fewer active latents keep all of them). Truncating further (e.g. to 512) reduces index size and latency. - **queries** are stored unpruned; truncate to the trained query budgets for faster search. ```python from datasets import load_dataset docs = load_dataset("lightonai/SlideOn-sparse-vectors", "beir-v1.0.0-scifact", split="corpus") k = 512 docs = docs.map(lambda r: {"indices": r["indices"][:k], "weights": r["weights"][:k]}) ``` ## Anserini `to_anserini.py` (in this repo) truncates to top-k, quantizes weights as `round(w * 100)` (the SPLADE convention; costs ≤ 0.001 nDCG@10 in our checks) and writes `JsonVectorCollection` documents and `TsvString` topics: ```bash python to_anserini.py beir-v1.0.0-scifact/corpus collections/scifact/part0.jsonl.gz --k 512 python to_anserini.py beir-v1.0.0-scifact/queries/test topics.scifact.tsv.gz --query bin/run.sh io.anserini.index.IndexCollection -collection JsonVectorCollection -input collections/scifact \ -index indexes/scifact.slideon -generator DefaultLuceneDocumentGenerator -threads 16 -impact -pretokenized bin/run.sh io.anserini.search.SearchCollection -index indexes/scifact.slideon -topics topics.scifact.tsv.gz \ -topicReader TsvString -output run.scifact.txt -impact -pretokenized -removeQuery -hits 1000 ``` ## Citation ```bibtex @misc{moura2026slideon, author = {Moura Junior, Paulo Roberto and Sourty, Raphaël}, title = {{SlideOn} \& {mSlideOn}: Sparsified late-interaction and dense retrievers}, year = {2026}, howpublished = {Hugging Face Blog}, url = {https://huggingface.co/blog/lightonai/slideon-mslideon} } ```