llm-jp/llm-jp-4-33b-thinking-dpo-data

llm-jp-4-33b-thinking-dpo-data

Overview

This dataset is a Direct Preference Optimization (DPO) dataset used to train llm-jp-4-33b-thinking.

It is constructed by pairing multiple candidate responses for a given prompt and selecting preferred (chosen) and non-preferred (rejected) responses. The splits reasoning_low, reasoning_medium, and reasoning_high correspond to different reasoning effort settings used during response generation.

The fields chosen_analysis, chosen_final, rejected_analysis, and rejected_final are generated using a model trained via supervised fine-tuning (SFT). These responses are then evaluated by gpt-oss-120b, which determines the preferred (chosen) and non-preferred (rejected) responses.

To support the continued development of LLM-jp, we would greatly appreciate it if you could share how you utilize LLM-jp outcomes via the survey form.

Data Sources and Licenses

The DPO dataset for llm-jp-4-33b-thinking is constructed from the following data sources. Since each dataset has its own license, please ensure compliance with the respective licenses when using this dataset. Some subsets cannot be redistributed due to licensing restrictions and are therefore not included.

Data Format

Each sample has the following format:

{
  "ID": "...",
  "messages": [
    {"role": "system", "name": null, "content": ["..."]},
    {"role": "user", "name": null, "content": ["..."]},
  ],
  "chosen_analysis": "...",
  "chosen_final": "...",
  "rejected_analysis": "...",
  "rejected_final": "..."
}

• —chosen_analysis represents the reasoning process (chain-of-thought) of the preferred response
• —chosen_final represents the final response of the preferred response
• —rejected_analysis represents the reasoning process of the non-preferred response
• —rejected_final represents the final response of the non-preferred response

The messages field contains the input conversation (prompt context). This structure allows training preference models that compare two candidate responses conditioned on the same prompt.

Notes

• —This dataset contains responses generated by a large language model.
• —The preference labels (chosen vs rejected) are constructed automatically and may contain noise.
• —Generated content may include errors, inaccuracies, biases, or harmful outputs.

Send Questions to

llm-jp(at)nii.ac.jp

Model Card Authors

The names are listed in alphabetical order.

Hirokazu Kiyomaru and Takashi Kodama.