tsanghasona/athena-corpus

---
license: mit
task_categories:
- text-retrieval
- feature-extraction
language:
- en
tags:
- philosophy
- literature
- religion
- semantic-search
- embeddings
- public-domain
size_categories:
- 100K<n<1M
---

# Athena Corpus

Pre-built data for [Athena](https://github.com/tsangha/athena), a semantic search engine for philosophy, literature, religion, and intellectual history.

## Contents

| File | Size | Description |
|------|------|-------------|
| `chunks.parquet` | 610 MB | 727k passages with text + metadata (author, work, tradition, era, genre, chapter, etc.) |
| `embeddings_bf16.npy` | 1.1 GB | 727k x 768 float16 vectors, L2-normalized ([jina-embeddings-v5-text-nano](https://huggingface.co/jinaai/jina-embeddings-v5-text-nano)) |
| `text/` | 1.6 GB | 5,012 cleaned source texts (plain text, one file per work) |

Rows in `chunks.parquet` and `embeddings_bf16.npy` are aligned by index -- row 0 in the parquet corresponds to row 0 in the embedding matrix.

## Corpus scale

- **4,980 works** from **2,310 authors** across **336 intellectual traditions**
- **727k chunks** (~640-768 tokens each)
- Spans ancient philosophy, sacred texts, literature, poetry, political theory, science, and more
- All texts are public domain

## Usage

### With Athena (full search server)

```bash
git clone https://github.com/tsangha/athena.git && cd athena
uv venv --python 3.11 && source .venv/bin/activate && uv pip install -e .

# Download this dataset
pip install huggingface_hub
hf download tsanghasona/athena-corpus --repo-type dataset --local-dir data/

# Export query encoder model
pip install torch transformers onnxsim
python embedder-rs/scripts/export_onnx.py --output-dir model/
python -m onnxsim model/model.onnx model/model_simplified.onnx

# Start server
uv run python -m uvicorn server.main:app --port 3003
```

### Standalone (just the data)

```python
import polars as pl
import numpy as np

# Load chunks with metadata and text
df = pl.read_parquet("chunks.parquet")
print(df.columns)
# ['text', 'chunk_id', 'author', 'work', 'text_type', 'tradition',
# 'era', 'genre', 'chapter', 'poem_title', 'chunk_index', ...]

# Load embeddings (memory-mapped for large files)
embeddings = np.load("embeddings_bf16.npy", mmap_mode="r")
print(embeddings.shape) # (726986, 768)
print(embeddings.dtype) # float16
```

## Embedding model

Embeddings were generated with [jina-embeddings-v5-text-nano](https://huggingface.co/jinaai/jina-embeddings-v5-text-nano):

- 768 dimensions, float16, L2-normalized
- Asymmetric: document chunks use the default prefix; queries should use `"Query: "` prefix
- Generated with Athena's Rust embedder (see repo for source)

## Chunking

Texts are split into ~640-768 token passages using strategy-appropriate chunking:

| Strategy | Used for | Examples |
|----------|----------|----------|
| `structured` | Numbered sections | Spinoza's Ethics, Aquinas's Summa |
| `discursive` | Flowing prose | Nietzsche, Plato's dialogues |
| `literary` | Chapter-aware paragraph merge | Dostoevsky, Homer, Tolstoy |
| `poetic` | Poem-boundary detection | Browning, Yeats, Heine |
| `annotation` | Markdown header split | Academic notes |

The `text_type` column in the parquet indicates which strategy was used for each chunk.

## License

MIT -- see the [Athena repo](https://github.com/tsangha/athena) for details.

All source texts are public domain.