llm-jp-4-8b-thinking-dpo-data

# llm-jp-4-8b-thinking-dpo-data ## Overview This dataset is a Direct Preference Optimization (DPO) dataset used to train [llm-jp-4-8b-thinking](https://huggingface.co/llm-jp/llm-jp-4-8b-thinking). It is constructed by pairing multiple candidate responses for a given prompt and selecting preferred (chosen) and non-preferred (rejected) responses. The splits `reasoning_low`, `reasoning_medium`, and `reasoning_high` correspond to different reasoning effort settings used during response generation. The fields `chosen_analysis`, `chosen_final`, `rejected_analysis`, and `rejected_final` are generated using a model trained via supervised fine-tuning (SFT). These responses are then evaluated by [gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b), which determines the preferred (chosen) and non-preferred (rejected) responses. To support the continued development of LLM-jp, we would greatly appreciate it if you could share how you utilize LLM-jp outcomes via the [survey form](https://forms.gle/AvbNXTNT2ADsssHq5). ## Data Sources and Licenses The DPO dataset for `llm-jp-4-8b-thinking` is constructed from the following data sources. Since each dataset has its own license, please ensure compliance with the respective licenses when using this dataset. Some subsets cannot be redistributed due to licensing restrictions and are therefore not included. | Dataset | License | Notes | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | ------------------------------------------------------- | | [ac_self_inst](https://huggingface.co/datasets/llm-jp/ac-self-inst) | answer-carefully-dataset-tou | Not included due to redistribution restrictions | | [daring_anteater](https://huggingface.co/datasets/nvidia/Daring-Anteater) | CC BY 4.0 | - | | [flan](https://huggingface.co/datasets/allenai/dolmino-mix-1124) | ODC-BY | - | | [ichikara](https://liat-aip.sakura.ne.jp/wp/llm%E3%81%AE%E3%81%9F%E3%82%81%E3%81%AE%E6%97%A5%E6%9C%AC%E8%AA%9E%E3%82%A4%E3%83%B3%E3%82%B9%E3%83%88%E3%83%A9%E3%82%AF%E3%82%B7%E3%83%A7%E3%83%B3%E3%83%87%E3%83%BC%E3%82%BF%E4%BD%9C%E6%88%90/) | - | Requires paid license, not included | | [jaster_v1.4.1](https://github.com/llm-jp/llm-jp-eval/tree/v1.4.1) | CC BY-SA 4.0, CC BY-SA 3.0, CC BY 4.0, BSD-3, Apache-2.0, MIT | - | | [llmjp_extraction_wiki_ja_v0.x](https://huggingface.co/datasets/llm-jp/extraction-wiki-ja) | Apache-2.0 | - | | [llmjp_magpie_sft_v1.0](https://huggingface.co/datasets/llm-jp/magpie-sft-v1.0) | Apache-2.0 | - | | llmc_math_dataset | - | To be released | | [logical_math_coding_wizard8x22b](https://huggingface.co/datasets/llm-jp/wizardlm8x22b-logical-math-coding-sft-ja) | Apache-2.0 | - | | [multiturn_calm3](https://huggingface.co/datasets/kanhatakeyama/AutoMultiTurnByCalm3-22B) | Apache-2.0, CC BY-SA 3.0, CC0, CC BY 4.0 | - | | [nemotron_post_v2_stem](https://huggingface.co/datasets/nvidia/Nemotron-Post-Training-Dataset-v2) | CC BY 4.0 | - | | [nemotron_post_v3_chat](https://huggingface.co/datasets/nvidia/Nemotron-Instruction-Following-Chat-v1) | CC BY 4.0 | - | | [nemotron_post_v3_if](https://huggingface.co/datasets/nvidia/Nemotron-Instruction-Following-Chat-v1) | CC BY 4.0 | - | | [nemotron_post_v3_math](https://huggingface.co/datasets/nvidia/Nemotron-Math-v2) | CC BY-SA 4.0 | - | | [random_to_fixed_multiturn_calm3](https://huggingface.co/datasets/kanhatakeyama/ramdom-to-fixed-multiturn-Calm3) | Apache-2.0 | - | | synthetic_if_ja | - | To be released | | [synthetic_jp_en_coding](https://huggingface.co/datasets/llm-jp/Synthetic-JP-EN-Coding-Dataset) | Apache-2.0 | - | ## Data Format Each sample has the following format: ```json { "ID": "...", "messages": [ {"role": "system", "name": null, "content": ["..."]}, {"role": "user", "name": null, "content": ["..."]}, ], "chosen_analysis": "...", "chosen_final": "...", "rejected_analysis": "...", "rejected_final": "..." } ``` * `chosen_analysis` represents the reasoning process (chain-of-thought) of the preferred response * `chosen_final` represents the final response of the preferred response * `rejected_analysis` represents the reasoning process of the non-preferred response * `rejected_final` represents the final response of the non-preferred response The `messages` field contains the input conversation (prompt context). This structure allows training preference models that compare two candidate responses conditioned on the same prompt. ## Notes * This dataset contains responses generated by a large language model. * The preference labels (chosen vs rejected) are constructed automatically and may contain noise. * Generated content may include errors, inaccuracies, biases, or harmful outputs. ## Send Questions to llm-jp(at)nii.ac.jp ## Model Card Authors The names are listed in alphabetical order. Hirokazu Kiyomaru and Takashi Kodama.