pairwise_dataset

PairWise Dataset for Disciplinary Depth AssessmentThis repository contains the **PairWise Dataset**, a labeled corpus designed to evaluate **disciplinary depth** in texts.The dataset consists of aligned pairs of introductory Wikipedia articles, where one text comes from the standard English Wikipedia and the other from Simple English Wikipedia. Each pair discusses the same concept while differing in linguistic and disciplinary complexity.## MotivationTo the best of our knowledge, no publicly available corpus specifically labels texts according to **disciplinary depth**. Existing resources on disciplinary literacy are either unavailable to the public or still under development.This dataset was created to support research on:- Disciplinary depth assessment- Text complexity analysis- Educational NLP- Readability modeling- Pairwise text classification---## Dataset ConstructionThe corpus was created by pairing articles from:- **English Wikipedia** (https://wikipedia.org)- **Simple English Wikipedia** (https://simple.wikipedia.org)Because both encyclopedias often contain articles describing the same concept, they provide naturally aligned text pairs with different levels of complexity.Simple English Wikipedia is intended for readers learning English or requiring easier language, using:- simpler vocabulary- shorter sentences- less technical terminologyThe regular Wikipedia articles generally contain more specialized vocabulary and more detailed explanations, making them appropriate examples of greater disciplinary depth.---## Data Collection PipelineThe dataset was extracted using the MediaWiki API through the following pipeline:1. Identify categories common to both Wikipedia editions.2. Remove overly generic or unsuitable categories.3. Apply automatic filtering to eliminate:- sensitive topics- inappropriate content- titles containing multiple words or numbers- profession-related pages and certain lexical patterns4. Retrieve up to 100 article titles per category.5. Verify that matching articles exist in both Wikipedias.6. Download the introductory extract of each article.7. Remove incomplete or missing pairs.8. Randomize the order of the two texts.9. Assign a binary label indicating which text is the more advanced version.The resulting dataset is balanced and covers a broad range of academic and scientific topics.---## Dataset FormatEach row contains one aligned pair of texts.| Column | Description ||---------|-------------|| `sentence_1` | First text in the pair || `sentence_2` | Second text in the pair || `label` | Binary label indicating which text is the advanced version || `category` | Topic category || `title` | Wikipedia article title |### Label Definition- `0` → `sentence_1` is the **Simple Wikipedia** version.- `1` → `sentence_1` is the **Regular Wikipedia** version.Because the order is randomized, models must infer which text demonstrates greater disciplinary depth.---## Example| sentence_1 | sentence_2 | label | category | title ||------------|------------|-------|----------|-------|| The palm-nut vulture... | The palm-nut vulture... | 1 | Vultures | Palm-nut vulture || Fulgurites are a rare type... | Fulgurites (from Latin...) | 0 | Glass | Fulgurite || A hand axe... | A hand axe is... | 1 | Anthropology | Hand axe |---## Characteristics- English language- Pairwise text classification- Balanced binary labels- Thousands of aligned Wikipedia article pairs- Broad disciplinary coverage- Automatically extracted using the MediaWiki API---## Intended UsesThe dataset can be used for research on:- disciplinary depth prediction- text complexity estimation- readability assessment- educational NLP- large language model evaluation- pairwise ranking and classification--- ## NotesThe dataset is distributed as standard CSV files.Some spreadsheet applications (particularly older versions of LibreOffice/OpenOffice) may not correctly interpret quoted text fields in the CSV files.The dataset was successfully opened and inspected using **Tablecruncher** on macOS.