praxis-benchmark-anon/praxis

---
license: cc-by-4.0
language:
- en
pretty_name: PRAXIS — Pragmatic Suggestion Mining Benchmark
size_categories:
- 100K<n<1M
task_categories:
- text-classification
- token-classification
tags:
- suggestion-mining
- pragmatics
- speech-acts
- review-analysis
- benchmark
- domain-heterogeneity
configs:
- config_name: gold_plus
data_files:
- split: train
path: data/gold_train.csv
- split: validation
path: data/gold_dev.csv
- split: test
path: data/gold_test.csv
- config_name: unified_pool
data_files:
- split: train
path: data/praxis_unified_v2.csv
- config_name: span_only
data_files:
- split: train
path: data/span_only_v2.csv
- config_name: llm_eval
data_files:
- split: full
path: data/llm_eval_subset.csv
- split: frontier
path: data/llm_eval_frontier_subset.csv
---

# PRAXIS — Pragmatic Suggestion Mining Benchmark

PRAXIS is the largest and most domain-diverse public benchmark for actionable
suggestion mining in user-generated reviews to date: **~174,860 labeled
instances spanning 68 industries** across consumer services, Amazon product
categories, and digital/media platforms, with a fully multi-annotator-verified
**15,000-row Gold+ test set** as the canonical evaluation set, a three-tier
pragmatic typology (form, affective modifier, specificity), and aligned
suggestion spans on every positive instance for label-plus-evidence
evaluation.

This is the data + code + metadata release accompanying our NeurIPS 2026
Evaluations & Datasets Track submission.

## Quick start

```python
import pandas as pd

# canonical evaluation set (15,000 rows, balanced 7,500/7,500, all 68 industries)
test = pd.read_csv("gold_test.csv")
print(test["macro_domain"].value_counts())
# Product 6858, Service 4422, Digital/media 2086, Hybrid retail 1634

# train / dev splits
train = pd.read_csv("gold_train.csv") # 13,526 balanced rows
dev = pd.read_csv("gold_dev.csv") # 3,388 balanced rows

# full labeled corpus (174,860 rows; for those who want to train on more
# data than the gold splits)
full = pd.read_csv("praxis_unified_v2.csv")

# span-only positives (16,683 rows, aligned spans, for evidence localisation)
spans = pd.read_csv("span_only_v2.csv")
```

## Files in this release

### Core dataset

| File | Rows | Columns | Description |
|---|---:|---:|---|
| `gold_test.csv` | 15,000 | 24 | **Canonical Gold+ test set.** Balanced 7,500 positive / 7,500 negative, covers all 68 industries (per-industry budget ≥ 100 except `Insurance` n=71 and `Tax/Accounting/Investment` n=66 which are coverage-saturated). |
| `gold_train.csv` | 13,526 | 23 | Balanced training split (6,763 positive / 6,763 negative), drawn from the unified pool with no overlap with `gold_test.csv` or `gold_dev.csv`. |
| `gold_dev.csv` | 3,388 | 23 | Balanced dev split (1,694 / 1,694) for hyperparameter selection. |
| `praxis_unified_v2.csv` | 174,860 | 22 | The **full PRAXIS labeled pool**. Includes all binary suggestion labels and, for positives, the three-tier typology + span. Use this if you want a corpus larger than gold_train. |
| `span_only_v2.csv` | 16,683 | 23 | Subset of positives that have a span localisable in the review text — supports the label-plus-evidence task. |
| `llm_eval_subset.csv` | 2,040 | 24 | Stratified subset used for the open / frontier LLM benchmarks in the paper (§"Frontier and large open-weight LLMs via a hosted cloud API"). Balanced 50/50, 30 rows per industry. |
| `llm_eval_frontier_subset.csv` | 544 | 24 | A smaller stratified subset (8 rows per industry) used for slow frontier LLMs to keep API budget within reason. |

### Predictions (per-row)

The bundle ships per-row predictions of every model in the leaderboard so
reviewers can recompute every reported cell:

```
predictions/
encoder/
bert-base-uncased/
test_predictions_seed{13,42,137}.csv
roberta-base/
test_predictions_seed{13,42,137}.csv
deberta-base/
test_predictions_seed{13,42,137}.csv
roberta-large/
test_predictions_seed{13,42,137}.csv
roberta-prauc/
aakash_roberta_test_predictions.csv (recall-weighted PR-AUC re-train of Trivedi et al. 2026)
llm/
ollama_gpt-oss_20b.csv
ollama_gpt-oss_120b.csv
ollama_ministral-3_8b.csv
ollama_ministral-3_14b.csv
ollama_nemotron-3-nano_30b.csv
ollama_gemma4_31b.csv
ollama_qwen3-next_80b.csv
ollama_kimi-k2-thinking.csv
ollama_deepseek-v4-flash.csv
ollama_deepseek-v4-pro.csv
ollama_glm-4.7.csv
leaderboard.json
leaderboard.csv
```

### Metadata

- `croissant.json` — Croissant 1.0 metadata with Responsible-AI fields.
- `DATASHEET.md` — Datasheet for Datasets (Gebru et al. 2021).
- `LICENSE.txt` — Annotation license: CC-BY-4.0 (review text subject to source-platform terms of use).

## Column dictionary (`gold_test.csv`, `gold_train.csv`, `gold_dev.csv`)

| Column | Type | Description |
|---|---|---|
| `gold_id` | string | Stable canonical row id (`gold_plus_NNNNN`). |
| `gold_split` | enum | `gold_plus_test` / `gold_plus_train` / `gold_plus_dev`. |
| `label` | int | Binary suggestion label (1 = positive, 0 = negative). |
| `is_suggestion` | bool | Same as `label` cast to bool. |
| `review_id` | string | Source-platform review id (Yelp business_id-style id, or row id from the Amazon source). |
| `industry` | string | One of 68 industry labels. |
| `macro_domain` | enum | `Service` / `Product` / `Digital/media` / `Hybrid retail`. |
| `text` | string | The full review text (whitespace-normalised). |
| `suggestion_span` | string | For positives, the verbatim suggestion span from the review (`NONE` for negatives). |
| `tier1_form` | enum | For positives: communicative form — `Direct imperative`, `Modal/deontic`, `Optative`, `Conditional`, `Interrogative`, `Comparative`. |
| `tier2_modifiers` | list | For positives: subset of `{Frustrated, Hedged, Backhanded, Sarcastic}` (multi-label, may be empty). |
| `tier3_specificity` | enum | For positives: `Vague` / `Specific` / `Very specific`. |
| `borderline` | bool | Whether the row is flagged as a borderline positive (low inter-annotator agreement). |
| `borderline_reason` | string | Free-text rationale for borderline cases. |
| `source_file` | string | Provenance: `progress.csv`, `negatives_verified.csv`, etc. |
| `source_row` | int | Row index in the source file. |
| `resolved_review_id` | string | Cleaned review id used for joining with full-text source. |
| `review_text_source` | string | Which source corpus the full text came from (`labelled_Reviews1_combined_5K_per_category.csv` or `yelp_reviews_specific.csv`). |
| `text_match_method` | string | How the row's text was looked up (e.g.\ `id_industry`). |
| `text_match_candidates` | int | Number of candidate matches in the source corpus. |
| `span_in_text` | bool | Whether the suggestion span is exactly substring-aligned to the review text. |
| `span_alignment_method` | string | `exact_normalized` / `not_applicable_negative` / etc. |
| `sampling_bucket` | string | The stratification bucket the row was sampled from. |
| `eval_weight_natural_binary` | float | Inverse-propensity weight for converting balanced-test-set scores back to a deployment-prevalence-weighted estimate. |

`praxis_unified_v2.csv` shares most columns; it additionally carries
`prediction_prob` (the production classifier's predicted probability),
`provenance` (high-level lineage), `has_text` (text-availability flag), and
the four boolean modifier columns (`mod_frustrated`, `mod_hedged`,
`mod_backhanded`, `mod_sarcastic`).

## How the data was constructed

Full details are in the paper's §3 (Dataset) and §3.2 (Annotation pipeline).
The reproducible chain-of-counts is:

### Source corpora

| Source file | Rows | Role |
|---|---:|---|
| `labelled_Reviews1_combined_5K_per_category.csv` | 155,000 | Upstream Amazon review corpus (~5K rows × 31 product/digital categories) |
| Yelp Open Dataset (service slice) | ~70K | Upstream service-review corpus |

### Stage-1 pre-filter (recall-oriented LLM)

| Output | Rows | Description |
|---|---:|---|
| `20K_positive.csv` | 20,650 | LLM-flagged candidate positives (high-recall, low-precision pre-filter) |
| `negatives_verified.csv` | 154,432 | LLM-flagged-negatives sent to annotators for false-negative auditing |

### Stage-2 human verification

| Outcome | Verified positives | Verified negatives |
|---|---:|---:|
| Among LLM-flagged positives (`progress.csv`, n=20,650) | **14,475** confirmed positive | 6,175 actually negative (pre-filter false-positive rate ≈ 30%) |
| Among LLM-flagged negatives (`negatives_verified.csv`, n=154,432) | **2,303** surfaced as actually positive (pre-filter false-negative rate ≈ 1.5%) | 152,129 confirmed negative |

### Stage-3 unified pool

After merging and deduplication:

| Metric | Count |
|---|---:|
| **Total rows** in `praxis_unified_v2.csv` | **174,860** |
| Verified positives | **16,683** |
| Verified negatives | **158,177** |
| Rows with full review text | 173,996 (99.5%) |
| Industries | **68** |
| Macro-domains | 4 (Product 114,991 · Digital/media 35,127 · Service 18,281 · Hybrid retail 6,461) |

### Stage-4 Gold+ stratification

A balanced industry-stratified Gold+ pool is carved out of the unified pool
for canonical evaluation. The Gold+ split totals are:

| Split | Total | Positives | Negatives |
|---|---:|---:|---:|
| `gold_test.csv` | **15,000** | **7,500** | **7,500** |
| `gold_train.csv` | 13,526 | 6,763 | 6,763 |
| `gold_dev.csv` | 3,388 | 1,694 | 1,694 |
| **Gold+ total** | **31,914** | **16,151** | **15,763** |

Stratification budget: ≥100 rows per industry where the pool size permitted;
the two smallest-pool industries (`Insurance` n=71, `Tax / Accounting /
Investment` n=66) are coverage-saturated.

### Stage-5 span alignment

Every positive's verbatim suggestion span is normalised and
substring-aligned to the review text; 99.9% of Gold+ positive spans align
exactly. The `span_only_v2.csv` file (16,683 rows) is the subset of all
verified positives in the unified pool with an aligned span — used for the
label-plus-evidence task.

## Splits

- `gold_test.csv` is the **canonical evaluation set**. Models in the
leaderboard report macro-F1 on this set.
- `gold_train.csv` and `gold_dev.csv` are the recommended training data;
we use them for all encoder fine-tunes in the paper.
- Models that want more training data may use `praxis_unified_v2.csv`,
but **must exclude rows whose `gold_id` is in `gold_test.csv` or
`gold_dev.csv`**, otherwise the leaderboard becomes uninterpretable.

## Citing

```
@inproceedings{praxis2026,
title = {PRAXIS: A Large-Scale, Multi-Domain Benchmark for Pragmatic Suggestion Mining in User-Generated Reviews},
author = {Anonymous},
booktitle = {Proceedings of the NeurIPS 2026 Evaluations and Datasets Track},
year = {2026},
}
```

## License

- **Annotations** (every column in this release except `text` and
`suggestion_span`): `CC-BY-4.0`.
- **Review text and suggestion-span snippets** (`text`, `suggestion_span`):
subject to the source-platform terms of use (Yelp Open Dataset License
for Yelp-derived rows; Amazon ToS for Amazon-derived rows). Users are
responsible for ensuring their downstream use respects those terms.

## Contact

For dataset bugs, leaderboard submissions, or other inquiries, open an issue
on the project's anonymised release repository (URL provided in the paper's
camera-ready version).