arXiv paper embeddings for DiRe topology-faithful dimensionality reduction

Paper-level mean-pooled embeddings for 723,457 arXiv papers, produced with BAAI/bge-small-en-v1.5 (384-d), together with arXiv descriptive metadata, 2-d and 3-d DiRe / UMAP layouts of the full corpus, and provenance manifests. This is the dataset underlying the arXiv experiment in: Kolpakov, A. and Rivin, I. DiRe: Topology-Faithful Dimensionality Reduction. (PNAS submission, 2026.) The pipeline is: 130 M LaTeX-derived chunks -> unit-normalized BGE-small chunk embeddings -> mean-pool per paper (one 384-d vector per paper) -> DiRe / UMAP projection. The released embeddings are the mean-pooled, un-renormalized vectors (norms ~0.83-1.0). Downstream code typically L2-renormalizes them before further use. Contents: embeddings/part-*.parquet (8 shards, ~1.03 GB) - arxiv_id: string, embedding: fixed_size_list<float32, 384> metadata.parquet - arXiv descriptive metadata (arxiv_id, title, primary_category, categories, n_chunks) layouts/layouts_2d.parquet - full-corpus 2-d layouts for DiRe and UMAP at n_neighbors in {8, 16, 32, 64, 128} layouts/layouts_3d.parquet - 3-d layouts at n_neighbors=16 for both methods provenance/ - embedding config, software versions, SHA-256 file hashes Source-text policy: this release contains only arXiv descriptive metadata (CC0 per arXiv API terms) and numerical research artifacts produced by the authors. Raw arXiv text, PDFs, TeX source, and chunk text are NOT redistributed. Users who want the full text of any individual paper should retrieve it directly from arXiv subject to that paper's license. Convenience mirror: huggingface.co/datasets/igriv/dire-arxiv-bge-small-embeddings Code: github.com/sashakolpakov/dire-rapids-arxiv (corpus pipeline) and github.com/sashakolpakov/dire-rapids (the DiRe reducer).