Zhaoming213/baiduwiki

--- license: apache-2.0 --- ## 数据集介绍 这是百度百科数据集，对原始数据集进行了简单清洗，最终变成了类似:{"text":....}对结构，jsonl文件。 ## 清洗代码 ``` import json input_file = "百度百科.jsonl" output_file = "filtered_data.jsonl" with open(input_file, 'r', encoding='utf-8') as infile, \ open(output_file, 'w', encoding='utf-8') as outfile: for line in infile: if not line.strip(): continue try: data = json.loads(line) outfile.write(json.dumps(data, ensure_ascii=False) + '\n') except json.JSONDecodeError: continue