msmarco_colbert

# MS MARCO ColBERT Embeddings Pre-computed ColBERT embeddings for [MS MARCO](https://huggingface.co/datasets/BeIR/msmarco) using [PyLate](https://github.com/lightonai/pylate) and [answerdotai/answerai-colbert-small-v1](https://huggingface.co/answerdotai/answerai-colbert-small-v1). ## Dataset Structure The dataset contains: - `data/corpus/`: 177 parquet files with document embeddings - `data/queries/`: 11 parquet files with query embeddings - `data/qrels/train.parquet`: Relevance judgments (532,751 pairs) ## Usage ```python from datasets import load_dataset # Load from directory (recommended for large datasets) corpus = load_dataset("parquet", data_dir="hf://datasets/WenxingZhu/msmarco_answerai_colbert_small_embeddings/data/corpus") queries = load_dataset("parquet", data_dir="hf://datasets/WenxingZhu/msmarco_answerai_colbert_small_embeddings/data/queries") qrels = load_dataset("parquet", data_files="hf://datasets/WenxingZhu/msmarco_answerai_colbert_small_embeddings/data/qrels/train.parquet") ``` ## Model Generated with [answerdotai/answerai-colbert-small-v1](https://huggingface.co/answerdotai/answerai-colbert-small-v1)