orionweller/tokenized-datasets-c4-msmarco-jsonl

--- dataset_info: - config_name: c4 features: - name: text dtype: string - name: id dtype: string splits: - name: train num_examples: 8596372 - config_name: msmarco features: - name: text dtype: string - name: id dtype: string splits: - name: train num_examples: 8596372 configs: - config_name: c4 data_files: - split: train path: c4/data-*-of-*.arrow - config_name: msmarco data_files: - split: train path: msmarco/data-*-of-*.arrow --- This dataset contains two configurations: c4 and msmarco, both stored in Arrow format with multiple sharded files.