arxiv_pool_daily

# Remyx Daily arXiv Pool Last 30 days of arXiv submissions in selected CS categories, with abstracts pre-embedded using `sentence-transformers/all-MiniLM-L6-v2` (384-dim). Backs the candidate pre-filter step in the [Remyx Paper Recommender Space](https://huggingface.co/spaces/remyxai/mhpd-paper-recommender). Pre-computed so the Space doesn't re-embed 500+ abstracts every cold start. ## Schema | Column | Type | Notes | |---|---|---| | `arxiv_id` | str | Without version suffix (e.g. `2605.23904`) | | `title` | str | One-line, newlines stripped | | `abstract` | str | Original LaTeX-stripped abstract | | `published_at` | str | `YYYY-MM-DD` | | `categories` | list[str] | arXiv categories (e.g. `["cs.AI", "cs.CL"]`) | | `abstract_embedding` | list[float32] | 384-dim, **NOT pre-normalized** — normalize at use time | ## Categories included Current run: `cs.AI`, `cs.LG`, `cs.CL`. **Known limitation**: this is AI/ML-heavy. Repos focused on adjacent domains (databases, distributed systems, computer vision) get weaker retrieval matches because their target papers aren't in the pool. Planned expansion: `cs.CV`, `cs.DB`, `cs.SE`, `stat.ML` in the next daily refresh. ## Refresh cadence Refreshed daily via `scripts/update_arxiv_pool.py` in the [mhpd-dpo-training repo](https://github.com/remyxai/mhpd-dpo-training/blob/main/space/scripts/update_arxiv_pool.py). Each refresh overwrites the parquet — older snapshots aren't retained. ## Usage ```python from datasets import load_dataset import numpy as np ds = load_dataset("remyxai/arxiv_pool_daily", split="train") embs = np.stack(ds["abstract_embedding"]) embs = embs / np.linalg.norm(embs, axis=1, keepdims=True) # normalize once # Query: embed your text with the same model, then cosine top-k from sentence_transformers import SentenceTransformer m = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2") q = m.encode("your team's domain summary", normalize_embeddings=True) sims = embs @ q top10 = np.argsort(-sims)[:10] for i in top10: print(ds[int(i)]["title"], sims[i]) ```