ai2-adapt-dev/tulu3.4-sft-replica-50k

---
configs:
- config_name: prompts_only
data_files:
- split: train
path: tulu3.4-sft-replica-50k.jsonl
- config_name: gpt4-prefs-on-policy
data_files:
- split: train
path: models_on-policy__preferences_gpt4.jsonl
- config_name: gpt4-prefs-off-policy
data_files:
- split: train
path: models_off-policy__preferences_gpt4.jsonl
- config_name: reference_prompts_off-policy
data_files:
- split: train
path: reference_prompts_off-policy.jsonl
- config_name: sft_vs_all
data_files:
- split: train
path: preferences_gpt4_sft-vs-all.jsonl
- config_name: math_vs_all
data_files:
- split: train
path: preferences_gpt4_math-vs-all.jsonl
- config_name: sft_vs_sft
data_files:
- split: train
path: preferences_gpt4_sft-vs-sft.jsonl
- config_name: math_vs_math
data_files:
- split: train
path: preferences_gpt4_math-vs-math.jsonl
- config_name: llama3.1-70b-prefs-on-policy
data_files:
- split: train
path: preferences_llama70b-on-policy.jsonl
---

# Tulu 3.4 SFT Replica 50k

I sampled ~50k instances from https://beaker.org/ds/01J7WZNKXSKRJJYQ1P0H9JFW04/details.
I tried to sample equally across datasets.
Useful for some preference experiments.

Looking for shards? *(213 shards with 250 rows each)* 💎

- Beaker: [ljm/tulu3.4-sft-replica-50k-shards](https://beaker.org/ds/01J9VSPJG4F4N8HBB396JBP2VC/details)
- AWS: `s3://ai2-ljm-dev/tulu3.4-sft-replica-50k/*.jsonl`

Looking for preferences? *(Prefix is always tulu3.4-sft-replica-50k-ultrafeedback-tpl THEN the model that judged it)*

- AWS: https://us-east-1.console.aws.amazon.com/s3/buckets/ai2-ljm-dev?region=us-east-1&bucketType=general&prefix=preferences/&showversions=false

Looking for the code? https://github.com/allenai/scaling-preferences

## Configs

Unless otherwise stated, the judge here is gpt-4-turbo-2024-04-09.

- gpt4-prefs-off-policy: generate responses based on 17 or so models similar to the Ultrafeedback (UF) paper. This configuration is the closest to applying the vanilla Ultrafeedback pipeline (minus the principle prompts) on our dataset.
- gpt4-prefs-on-policy: we add responses from the (i) Tulu 3.4 SFT and (ii) L3.1* math checkpoints. We pair models as usual so these on-policy checkpoints might just be a small proportion of all matchups.
- sft_vs_all: one response was generated solely by the Tulu 3.4 SFT checkpoint while the other response was generated by any of the 17+ models in the UF paper.
- math_vs_all: one response was generated solely by the L3.1* math checkpoint while the other response was generated by any of the 17+ models in the UF paper.
- sft_vs_sft: both responses were generated by the Tulu 3.4 SFT checkpoint.
- math_vs_math: both responses were generated by the Math L3.1 checkpoint.

## Dataset Breakdown

| dataset | count |
|:------------------------------------------------------|--------:|
| 0 | 3636 |
| AI-MO/NuminaMath-TIR | 3636 |
| HuggingFaceH4/no_robots | 3636 |
| WizardLMTeam/WizardLM_evol_instruct_V2_196k | 3636 |
| ai2-adapt-dev/Daring-Anteater-reformat | 3636 |
| ai2-adapt-dev/SlimOrca-reformat | 3636 |
| ai2-adapt-dev/WebInstructSub-reformat | 3636 |
| m-a-p/CodeFeedback-Filtered-Instruction | 3636 |
| ai2-adapt-dev/WildChat-1M-Full-GPT4-Only | 3636 |
| ai2-adapt-dev/aya_dataset-reformat | 3636 |
| allenai/openassistant-guanaco-reformatted | 3636 |
| ai2-adapt-dev/Table-GPT-All-train | 3000 |
| science.scientific_papers_summarization_single_doc | 578 |
| science.data_reco_mcq | 555 |
| science.chemprot | 532 |
| science.craftchem | 309 |
| science.scicite | 307 |
| science.covid_deepset | 302 |
| science.acl_arc_intent | 301 |
| science.healthver | 298 |
| science.bioasq_general | 294 |
| science.bioasq_factoid | 292 |
| science.chemsum_single_document | 292 |
| science.drug_combo_extraction | 290 |
| science.bc7_litcovid_topic | 289 |
| science.chia | 285 |
| science.scientific_lay_summarisation_plos_single_doc | 285 |
| science.medmentions | 284 |
| science.mslr2022_ms2_multidoc | 281 |
| science.scitldr | 280 |
| science.qasa_abstractive | 279 |
| science.mslr2022_cochrane_multidoc | 273 |
| science.scientific_lay_summarisation_elife_single_doc | 271 |
| science.scireviewgen_multidoc | 271 |
| science.multixscience_multidoc | 265 |
| science.bioasq_yesno | 264 |
| science.genia | 260 |
| science.pico | 256 |
| science.chemdner | 247 |
| science.covidfact | 235 |
| science.qasper_extractive | 178 |
| science.ddi | 177 |
| science.ncbi | 176 |
| science.anat_em | 170 |
| science.cdr | 143 |
| science.pubmedqa | 133 |
| science.nlmgene | 103 |
| science.gnormplus | 96 |
| science.scierc | 95 |
| science.annotated_materials_syntheses | 45 |
| hard_coded | 14 |
| science.linnaeus | 4 |
| science.mltables | 3 |
| science.chemtables | 1 |
| science.nlmchem | 1 |