middle_training

# middle_training This dataset is a globally deduplicated middle-training raw-text subset converted from BAAI IndustryCorpus2. It is intended for continued pretraining / middle training of small language models, especially when the goal is to improve general text understanding before downstream instruction tuning. ## Summary - Final records after global deduplication: **1,021,694** - LLM-kept source records before global deduplication: **1,077,123** - Removed duplicates: **55,429** - Duplicate by source `_id`: **26,013** - Duplicate by normalized text hash: **29,416** - Empty text removed: **0** - Languages: Chinese **837,467** / English **184,227** - LLM quality buckets: high **471,064** / medium **550,630** ## Source And Pipeline The source data comes from multiple BAAI IndustryCorpus2 sub-datasets on ModelScope. The processing pipeline was: 1. Sample raw candidates from IndustryCorpus2 parquet shards by industry category and language. 2. Apply metadata and text-quality filters before LLM filtering. 3. Use a local vLLM service with `Qwen3-4B-Instruct-2507` to judge whether each candidate is suitable for 0.6B-scale middle training. 4. Keep only LLM-approved `high` and `medium` quality rows. 5. Run global deduplication across all kept rows using source `_id` and normalized text hash. 6. Export both normalized raw-text files and ms-swift pretraining files. ## Cleaning Rules The first-stage deterministic cleaning used IndustryCorpus2 metadata and simple text statistics. Default Chinese rules: - `perplexity <= 800` - `quality_score >= 3.5` - text length between `80` and `900` characters - `word_rep_ratio <= 0.10` - `char_rep_ratio <= 0.04` - `special_char_ratio <= 0.20` - `alnum_ratio >= 0.78` Default English rules: - `quality_score >= 3.5` - text length between `80` and `2500` characters - output text truncated to at most `700` characters when configured - `word_rep_ratio <= 0.15` - `char_rep_ratio <= 0.10` - `special_char_ratio <= 0.35` - `alnum_ratio >= 0.70` - English perplexity was not used because many English shards do not provide reliable `perplexity`. Some categories used stricter overrides. For example, Film & Entertainment Chinese data used `perplexity <= 900`, `max_chars <= 800`, `char_rep_ratio <= 0.03`, `special_char_ratio <= 0.18`, and `alnum_ratio >= 0.80`. ## LLM Filtering Rules The second-stage semantic filter used a local vLLM deployment of `Qwen3-4B-Instruct-2507`. Rows were kept if they were: - natural, fluent, and coherent; - relevant to the target industry category; - suitable as ordinary raw text for continued pretraining; - article-like, explanatory, descriptive, review-like, or service-information-like text; - not overly noisy or specialized for a small model. Rows were dropped if they were: - download pages, pirated resource pages, cloud-disk pages, BT/magnet/free-full-version pages; - job posts, resumes, or job-duty pages unless the category explicitly benefits from employment information; - overly sensitive gossip, pornographic, violent, suicide-related, or crime-detail text; - large lists, ads, marketing spam, SEO stitching, garbled text, severe truncation, or heavy repetition; - clearly off-topic for the industry; - too specialized or strange and likely to increase perplexity for a 0.6B model. The LLM returned a structured judgment with `keep`, `quality`, `content_type`, `reason`, and `issue_tags`. Only kept rows were exported. ## Global Deduplication Deduplication was applied after all category-level LLM filtering: - First key: stable source ID from `raw_id` or `meta._id`. - Second key: SHA1 of normalized text. - Text normalization: collapse whitespace, trim, lowercase, then hash. - If either source ID or normalized text hash was already seen, the row was removed. This removed **55,429** duplicated rows from **1,077,123** LLM-kept rows, leaving **1,021,694** final records. ## Data Modules | Module | Records | | --- | ---: | | `raw_text` | 1,021,694 | | `ms_swift_pretrain` | 1,021,694 | `raw_text/*.jsonl` contains normalized records with metadata and LLM filter explanations. `ms_swift_pretrain/*.jsonl` contains ms-swift pretraining format: ```json {"messages": [{"role": "assistant", "content": "..."}]} ``` ## Category Counts | Category | Slug | Records | | --- | --- | ---: | | Subject Education | `subject_education` | 97,620 | | Finance & Economics | `finance_economics` | 88,902 | | Health & Medicine | `health_medicine` | 87,891 | | Transportation | `transportation` | 84,145 | | Film & Entertainment | `film_entertainment` | 81,443 | | Travel & Geography | `tourism_geography` | 72,322 | | Hospitality & Catering | `hospitality_catering` | 59,742 | | Agriculture & Fisheries | `agriculture_fisheries` | 51,590 | | Automobiles | `automobile` | 50,633 | | Sports | `sports` | 50,158 | | Politics & Administration | `politics_administration` | 45,343 | | Manufacturing | `manufacturing` | 44,656 | | Computing & Telecommunications | `computing_telecom` | 44,306 | | Real Estate & Construction | `real_estate_construction` | 42,647 | | Media & Journalism | `media_journalism` | 38,715 | | Law & Justice | `law_judiciary` | 36,124 | | Safety Management | `safety_management` | 13,554 | | Information Services | `information_services` | 9,572 | | Literature & Emotions | `literature_emotions` | 9,271 | | Water Resources & Marine | `water_marine` | 4,772 | | Power & Energy | `power_energy` | 4,573 | | Gaming | `gaming` | 3,715 | ## Files - `raw_text/raw_text_*.jsonl`: normalized raw text records. - `ms_swift_pretrain/pretrain_*.jsonl`: ms-swift pretraining records. - `dataset_info.json`: full counts by language, category, slug, source file, quality, and dedupe statistics. - `DATA_CLEANING.md`: detailed cleaning and filtering documentation. ## Raw Text Row Schema Raw text rows contain: ```json { "type": "raw_text", "source": "modelscope:BAAI/IndustryCorpus2_xxx", "category": "...", "industry_type": "...", "slug": "...", "language": "chinese|english", "quality_bucket": "high|middle", "llm_quality": "high|medium", "content_type": "...", "text": "...", "meta": { "_id": "...", "perplexity": 123.4, "quality_score": 4.0, "num_words": 100, "alnum_ratio": 0.9, "special_char_ratio": 0.1, "char_rep_ratio": 0.0, "word_rep_ratio": 0.0 }, "filter_reason": "...", "issue_tags": [] } ``` ## Example ms-swift Usage ```bash swift sft --dataset /path/to/ms_swift_pretrain/*.jsonl ... ```