bezhanidze/PARCOMED_research_only

PARCOMED（PARTAGES 开放医学文档语料库）是一个专门为研究和商业使用设计的法语生物医学文本数据集。该数据集旨在解决法语生物医学数据稀缺的问题，以增强大型语言模型在医学领域的多语言能力。它包含来自多种来源的文档，如临床病例、医学对话、教育资料、百科全书条目、医学文献、药物信息、问答对、法规文本和科学出版物等。数据集经过严格的许可审查，确保合规性，并经过预处理，包括文本清洗（如Unicode转换、URL去除）和去重（基于MinHash相似性）。数据集提供两个配置：finetuning（用于微调任务，包含905,342个样本，特征包括input、source和document_type）和instruction-tuning（用于指令调优任务，包含22,390个样本，额外特征包括instruction和output）。文档类型涵盖临床、对话、教育、百科全书、医学、药物、问答、法规和科学等类别。数据来源包括E3C、CAS、FRASIMED、PXCORPUS、WIKIPEDIA、EMEA_V3等多个公开数据集。统计信息显示，finetuning配置中科学类文档最多（640,313个），而instruction-tuning配置主要集中在问答类文档（22,390个）。数据集以parquet格式组织，便于使用HuggingFace的load_dataset函数加载。该数据集由PARTAGES开发团队协作创建，适用于自然语言处理任务，如模型微调、指令调优和医学文本分析。

PARCOMED (PARTAGES Corpus of Open MEdical Documents) is a French biomedical text dataset designed for research-only and commercial use. It addresses the scarcity of French biomedical data to enhance the multilingual capabilities of large language models in the medical domain. The corpus comprises documents from diverse sources, including clinical cases, medical dialogues, educational materials, encyclopedic entries, medical texts, medicinal information, question-answering pairs, regulatory documents, and scientific publications. It undergoes rigorous licensing scrutiny for compliance and preprocessing steps such as text cleaning (e.g., Unicode normalization, URL removal) and deduplication (using MinHash similarity). The dataset offers two configurations: finetuning (for fine-tuning tasks, with 905,342 samples featuring input, source, and document_type) and instruction-tuning (for instruction-tuning tasks, with 22,390 samples adding instruction and output features). Document types span clinical, dialogue, education, encyclopedic, medical, medicinal, question answering, regulation, and scientific categories. Sources include publicly available datasets like E3C, CAS, FRASIMED, PXCORPUS, WIKIPEDIA, and EMEA_V3. Statistics show that the finetuning configuration is dominated by scientific documents (640,313 samples), while instruction-tuning focuses on question-answering documents (22,390 samples). Organized in parquet format, it is easily loadable via HuggingFaces load_dataset function. Created collaboratively by the PARTAGES development team, it supports NLP tasks such as model fine-tuning, instruction-tuning, and medical text analysis.