RexiaAI/rexia-synthetic-pretrain-1m

--- license: apache-2.0 language: - en tags: - instruction-tuning - chat - synthetic - conversational - pretrain size_categories: - 1M<n<10M task_categories: - text-generation dataset_info: features: - name: text dtype: string - name: source dtype: string splits: - name: train num_bytes: 1147064728 num_examples: 984372 download_size: 680967212 dataset_size: 1147064728 configs: - config_name: default data_files: - split: train path: data/train-* --- # Rexia Synthetic Pretrain 1M A synthetic instruction-following dataset of ~984k cleaned, deduplicated conversations generated using Ministral-3B (Apache 2.0 licence) across 9 diverse categories. Designed as a large-scale pre-training / continual pre-training corpus for small language models, providing high-quality, stylistically varied data that avoids the GPT-4 stylistic bias common in datasets such as OpenHermes and SlimOrca. This is the larger companion to [RexiaAI/rexia-synthetic-chat-500k](https://huggingface.co/datasets/RexiaAI/rexia-synthetic-chat-500k), roughly doubling the sample count. --- ## Dataset Details | Category | Samples | |---|---| | Conversational / advice / opinions | ~205,000 | | Factual Q&A | ~195,000 | | Concept explanation | ~160,000 | | Step-by-step reasoning | ~145,000 | | Mathematics word problems | ~89,000 | | Creative writing | ~79,000 | | Comparison / analysis | ~70,000 | | Multi-turn dialogue | ~40,000 | | Coding (Python, JS, SQL, Bash, etc.) | ~1,800 | | **Total** | **~984,000** | --- ## Format Each sample contains a `text` field formatted for instruction tuning: ``` <|user|> {question} <|assistant|> {answer}<|end|> ``` Multi-turn samples include multiple exchanges: ``` <|user|> {question_1} <|assistant|> {answer_1}<|end|> <|user|> {question_2} <|assistant|> {answer_2}<|end|> ``` A `source` field identifies the category (e.g. `synthetic_coding`, `synthetic_factual`). --- ## Generation - **Generator model:** `ministral-3b` via Ollama (Apache 2.0 licence) - **Parallelism:** Multiple concurrent workers - **Cleaning pipeline:** - Encoding artefact removal - Quality filtering (minimum length, refusal detection, alpha-character ratio, repetition check) - Exact hash deduplication - MinHash LSH near-deduplication (Jaccard threshold 0.82, 128 permutations, 5-gram shingles) --- ## Intended Use Pre-training or continual pre-training of small language models (100M–3B parameters) for general instruction following and conversational ability. The large volume and diverse category coverage help models learn broad knowledge and varied response styles before targeted fine-tuning. --- ## Relationship to rexia-synthetic-chat-500k | Dataset | Samples | Primary use | |---|---|---| | [rexia-synthetic-chat-500k](https://huggingface.co/datasets/RexiaAI/rexia-synthetic-chat-500k) | ~487k | Fine-tuning / SFT | | **rexia-synthetic-pretrain-1m** | **~984k** | Pre-training / continual pre-training | --- ## Licence Apache 2.0 — generated from Ministral-3B which is Apache 2.0 licensed.