openthoughts4-code-9168-prompts-qwen3-30b-a3B-thinking-2507-n16-flattened-logprobs-k16

# OpenThoughts-4 Code SDG: Qwen3-30B-A3B-Thinking-2507 (n=16, top-16 logprobs) Synthetic generations from [**Qwen/Qwen3-30B-A3B-Thinking-2507**](https://huggingface.co/Qwen/Qwen3-30B-A3B-Thinking-2507) on the [Marin OpenThoughts-4 code SDG prompt set](https://huggingface.co/datasets/marin-community/hero-run-4-code-sdg-prompts-python-fenced-n16). Each prompt is sampled `n=16` times, and for every generated token the dataset stores the chosen-token log probability plus the **top-16** log probabilities over the vocabulary, enabling distillation, KL-style fine-tuning, reranking, and uncertainty analysis. ## Generation setup | Field | Value | |---|---| | Generator model | `Qwen/Qwen3-30B-A3B-Thinking-2507` | | Source prompts | `marin-community/hero-run-4-code-sdg-prompts-python-fenced-n16` (9,168 unique prompts) | | Samples per prompt (`n`) | 16 | | Logprobs returned (`k`) | 16 (top-k vocab logprobs per generated token) | | Max generated tokens | 32,768 | | Max model length | 34,816 | | Temperature | 0.8 | | Inference engine | vLLM on TPU v6e-4, `tensor_parallel_size=4` | | Producer | [marin-community/marin](https://github.com/marin-community/marin) — `experiments/sdg/code/qwen3-30b-a3b-thinking-2507/sdg_ot4_30k_code_qwen3_30b_a3b_thinking_32768_tokens.py` | ## Schema The dataset is a flattened parquet table with **one row per `(prompt, sample_index)`** pair. With 9,168 unique prompts and `n=16` samples each, the dataset contains 146,688 rows total. ### Identifier columns | Column | Type | Description | |---|---|---| | `prompt_index` | int64 | 0-based index of the source prompt within the prompt set (0 … 9,167) | | `response_index` | int64 | 0-based sample index within that prompt (0 … 15) | | `_unique_row_id` | string | Stable globally-unique id for the `(prompt, response)` pair, copied from the source prompt set; safe key for joins and dedup | | `instruction_seed` | string | The original natural-language coding problem from OpenThoughts-4 (before any chat templating) | | `generation_prompt` | string | The chat-templated prompt actually fed to vLLM (Qwen3 chat template applied to `instruction_seed`) | A given prompt is fully identified by either `prompt_index` or `_unique_row_id`; the 16 samples for a prompt share both keys and differ only in `response_index`. ### Generation columns | Column | Type | Description | |---|---|---| | `generated_text` | string | Decoded model response (everything after the chat template's assistant turn) | | `generated_token_ids` | list[int32], length `T` | Token ids of the generated response | | `generated_token_logprobs` | list[float32], length `T` | Log probability of each chosen token under the model | | `generated_top_logprob_token_ids` | list[int32], length `T × k` | **Flattened** top-k candidate token ids at each step | | `generated_top_logprobs` | list[float32], length `T × k` | **Flattened** log probabilities of those top-k candidates at each step | Where `T` = number of generated tokens for that row and `k = 16` (the top-k logprobs setting). ### Reshaping the flattened top-k arrays vLLM was run with `flat_logprobs=True` for compactness, so the per-step top-k arrays are stored as 1-D lists. To recover the standard `(T, k)` shape: ```python import numpy as np import pyarrow.parquet as pq df = pq.read_table("part-000000.parquet").to_pandas() row = df.iloc[0] T = len(row["generated_token_ids"]) k = 16 top_ids = np.asarray(row["generated_top_logprob_token_ids"]).reshape(T, k) top_logps = np.asarray(row["generated_top_logprobs"]).reshape(T, k) # top_ids[t, :] = top-16 candidate token ids at step t # top_logps[t, :] = log p of those candidates at step t (sorted high → low by vLLM) # the chosen token may or may not appear in the top-k; use generated_token_logprobs # for the chosen-token logprob. ``` ### File layout The dataset is sharded into `~2,900` parquet files named `part-NNNNNN.parquet`, each with up to 64 rows, totalling **~128 GB**. Use the `default` config to load the full `train` split. ## Companion datasets | Slice | Generator | |---|---| | code | Qwen3-30B-A3B-Thinking-2507 (this dataset) | | code | [Qwen3-32B](https://huggingface.co/datasets/marin-community/openthoughts4-code-9168-prompts-qwen3-32b-n16-flattened-logprobs-k16) | | code | [Qwen3-4B](https://huggingface.co/datasets/marin-community/openthoughts4-code-9168-prompts-qwen3-4b-n16-flattened-logprobs-k16) | | code | Gemma-4-31B-IT *(forthcoming)* | | science | Qwen3 family + Gemma-4-31B-IT *(forthcoming)* | ## License Released under Apache 2.0. The underlying generator model (`Qwen/Qwen3-30B-A3B-Thinking-2507`) is governed by its own license; consult the model card before redistribution. ## Citation If you use this dataset, please cite the [Marin project](https://github.com/marin-community/marin) and the [Qwen3 technical report](https://arxiv.org/abs/2505.09388).