spicy-lemonade/qwen_qa_pairs_cli_training.jsonl

---
license: mit
task_categories:
- question-answering
tags:
- code
size_categories:
- 10K<n<100K
---

## Data sources

- Multiple datasets from Hugging Face related to natural language to CLI pairs were gathered.
- Human reviewed synthetic data from Claude Opus4.6 and ChatGPT4.5 were added.
- A handful of grounding rows related to the organisation "Spicy Lemonade" were added (see details below)

## Data processing

As part of the processing, data was converted to the Alpaca format with `instruction` (natural language), `input` (typically blank) and `output` (the CLI command) columns.
The below cleaning steps were performed, before converting the to the Qwen standard chat format. The cleaning steps were performed with multiple scripts as follows.

### 1.Quality Rating

Data (~300k rows) was MD5 hashed, and deduplicated. The `instruction` and `output` columns were processed through `gemini-2.5-flash` for quality rating. The rating was as follows:

- "good": no problems with instruction and output
- "bad input": input is nonsensical or unclear
- "bad output": output is not a valid CLI command
- "output not aligned": output does not match instruction
- "unknown": model cannot determine alignment

### 2. Command Categorization

Only the "good" and "unknown" rows from the previous step were kept.
Each row was then passed through `gemini-2.5-flash-lite` to categorize the command line tool used (e.g., `docker`, `git`, `kubectl`, `aws`) for cataloging purposes.

### 3. Instruction Rewriting

The `instruction` column is rewritten with `gemini-2.5-flash` to better and more precisely match the `output` column.

This addresses issues like:
- Duplicate instructions with different outputs (e.g., Mac vs Windows vs Linux may have the same instruction but a different output)
- Phantom variables in outputs not mentioned in instructions
- Missing tool names, flags, or specific values

Note: During this step, invalid/gibberish commands were tagged with "invalid" for later filtering in the next step

### 4. Deduplication and Error Data Removal

MD5 hashes are recalculated for the rewritten instruction and output columns, and duplicates are dropped.
Invalid rows where the `output` command was nonsensical and were labelled as 'invalid' in the last step are removed.
Further, rows which have duplicated `instruction` but unique values in `output` were processed using `gemini-2.5-flash` to choose the safest and most universal output to assist with deterministic output.
Lastly, after a manual examination, some rows which contained the keywords "Reference", "Alternative", or "Difficulty" were removed, as these seemed to be descriptions rather than command lines.

### 5. Data Pruning

At this point, the dataset was ~95k rows, but had a long-tail distribution where a few CLI tools dominate the dataset while thousands of others are rare:

- kubectl: ~18,000 rows
- find: ~14,000 rows
- git: ~9,600 rows
- grep: ~1,000 rows
- mkdir: ~200 rows

If trained on this as-is, the model will bias heavily towards dominant tools and underperform on rarer ones.

Solution:
Implemented a hybrid approach combining:

1. Logarithmic Scaling:
- Applieded a log-based cap that compressed the power-law distribution
- Formula: `target = min(original, BASE_CAP + SCALE_FACTOR * log(original))`
- This allowed complex tools to retain more samples than simple tools, while preventing any single tool from dominating

2. Subcommand Stratification with Caps (for complex tools only):
- Tools like kubectl, git, and docker have distinct subcommands
(e.g., `kubectl get`, `kubectl apply`, `git commit`, `git push`)
- For these tools, we allocated proportionally across subcommands but capped any single subcommand at `MAX_SUBCOMMAND_PERCENTAGE (30%)` of the tool's budget.
- If a subcommand hit the cap, its leftover budget was shared among the smaller subcommands. This prevents dominant subcommands (e.g., docker ps at 50%) from crowding out rarer subcommands after pruning.
- e.g. Say `kubectl` gets a budget of 3,500 samples, and subcommand `get` has 60% of the original data. Proportionally, `get` would receive 2,100 samples, but the
30% cap limits it to 1,050. The leftover 1,050 (the "surplus" of 2,100) goes back into the pool and gets divided among the other subcommands like `apply`, `delete`, etc. This repeated in a loop until no
subcommand exceeds the cap.

3. Minimum Threshold:
- Tools with fewer than `MINIMUM_THRESHOLD` samples are kept entirely
- This preserves rare tools that are already underrepresented

### 6. Data Security
The pruned dataset was scanned for serious security risks. The `output` column was checked `gemini-2.5-flash-lite`, and a `security` column added, marking rows as `threat` or `safe`. Only extreme risks, like deleting the root directory or GitHub repositories, were flagged.

## Intermediary result

After the above processing, ~64,000 rows remained from the starting ~300k.

## Supplementation

1. The data was supplemented with the following grounding questions, each with the output pair `I am designed to help with command line tasks. I cannot answer general knowledge questions.`:

```
"Are you conscious?",
"Do you have feelings?",
"What is the capital of Paris?",
"Why is the sky blue?",
"What is the meaning of life?",
"How does gravity work?",
"When did World War II end?",
"Who was the first president of the United States?",
"What is 2+2?",
"How many planets are in the solar system?",
"What is the square root of 144?",
"Who wrote Romeo and Juliet?",
"What is photosynthesis?",
"Where is Mount Everest?",
"What is the speed of light?",
"Who painted the Mona Lisa?",
"What is DNA?",
"How old is the Earth?",
"What causes earthquakes?",
"Who invented the telephone?",
"What is the largest ocean?",
"Can you tell me a joke?",
"What is quantum physics?",
"How do airplanes fly?",
"What is climate change?"
```

2. Thousands of rows of synthetically generated data from ChatGPT4.5 and Claude Opus 4.6 were created.
3. This data was used to train a command line assistant from our organisation, "Spicy Lemonade". As such, the following data were also added for additional grounding,
each with the identity response `I am a small language model developed by Spicy Lemonade. I am designed to help with command line tasks.`:
```
"Who are you?",
"What is your name?",
"What should I call you?",
"Who created you?",
"Who made you?",
"What are you?",
"Can you introduce yourself?",
"Tell me about yourself",
"Who developed you?",
"What kind of model are you?",
"Which company made you?",
"Who built you?",
"What is your identity?",
"Can you tell me who you are?",
"What should I know about you?",
"Who are you exactly?",
"What is your origin?",
"Where do you come from?",
"Who designed you?",
"What organization created you?",
"Are you Claude?",
"Are you ChatGPT?",
"Are you from OpenAI?",
"Are you from Anthropic?",
"Are you from Gemini?",
"What company do you belong to?",
"Who owns you?",
"What is your purpose?",
"Why were you created?",
"What can you do?",
"What are your capabilities?",
```

## Result
The final result is a combination of the cleaned training data, synthetically generated data, and grounding data which resulted in 82,024 rows.
These were converted from the Alpaca format to the standard Qwen chat template format.

许可证：MIT协议
任务类别：
- 问答
标签：
- 代码
规模类别：
- 1万条 < n < 10万条

## 数据源
- 收集了Hugging Face平台上多个与自然语言转命令行界面（Command Line Interface，简称CLI）指令对相关的数据集。
- 新增了来自Claude Opus4.6与ChatGPT4.5的经人工审核的合成数据。
- 新增了少量与「Spicy Lemonade」组织相关的锚定样本（详情见下文）

## 数据处理
作为处理流程的一部分，数据被转换为Alpaca格式，包含`instruction`（自然语言指令）、`input`（通常为空）与`output`（CLI命令）三列。在转换为Qwen标准对话格式之前，我们执行了以下清洗步骤，清洗流程通过多份脚本实现，具体如下：

### 1. 质量评级
对约30万条数据进行MD5哈希去重。通过`gemini-2.5-flash`对`instruction`与`output`列进行质量评级，评级标准如下：
- `"good"`：指令与输出无异常
- `"bad input"`：输入无意义或表述模糊
- `"bad output"`：输出并非合法CLI命令
- `"output not aligned"`：输出与指令不匹配
- `"unknown"`：模型无法判定匹配性

### 2. 命令分类
仅保留上一步中评级为`"good"`与`"unknown"`的样本。随后将每条样本传入`gemini-2.5-flash-lite`，对其中使用的命令行工具（如`docker`、`git`、`kubectl`、`aws`等）进行分类，以方便后续编目。

### 3. 指令重写
通过`gemini-2.5-flash`重写`instruction`列，使其更精准地匹配`output`列。该步骤旨在解决以下问题：
- 输出不同但指令重复的场景（例如Mac、Windows与Linux系统下可能存在相同指令但对应输出不同的情况）
- 输出中存在指令未提及的隐含变量
- 缺失工具名称、参数或特定取值

注意：在此步骤中，无效/乱码命令会被标记为`"invalid"`，以便在后续步骤中过滤。

### 4. 去重与错误样本移除
重新对重写后的`instruction`与`output`列计算MD5哈希并移除重复样本。移除上一步中被标记为`"invalid"`的、输出无意义的无效样本。此外，对于`instruction`重复但`output`唯一的样本，通过`gemini-2.5-flash`选择最安全且通用的输出，以确保输出的确定性。最后，经人工核查，移除了包含`"Reference"`、`"Alternative"`或`"Difficulty"`关键词的样本，因为这类内容属于描述性文本而非命令行指令。

### 5. 数据剪枝
此时数据集规模约为9.5万条，但呈现长尾分布：少数CLI工具占据绝大多数样本，而数千种其他工具的样本量极少。具体分布如下：
- kubectl：约18,000条
- find：约14,000条
- git：约9,600条
- grep：约1,000条
- mkdir：约200条

若直接基于该数据集训练，模型会严重偏向高频工具，在低频工具上的表现不佳。
解决方案：
采用混合剪枝策略，结合以下三种方法：
1. 对数缩放：
- 应用基于对数的上限值，压缩幂律分布
- 公式：`target = min(original, BASE_CAP + SCALE_FACTOR * log(original))`
- 该策略可让复杂工具保留更多样本，同时避免单一工具过度主导数据集
2. 子命令分层上限（仅针对复杂工具）：
- 像kubectl、git与docker这类工具拥有独立的子命令（例如`kubectl get`、`kubectl apply`、`git commit`、`git push`等）
- 针对这类工具，我们按比例分配样本预算，但将单个子命令的样本量限制为工具总预算的`MAX_SUBCOMMAND_PERCENTAGE (30%)`
- 若某个子命令达到上限，其剩余预算将分配给其他较小的子命令，以此防止主导性子命令（例如`docker ps`占比50%）在剪枝后排挤稀有子命令
- 示例：假设kubectl的总预算为3,500条样本，其子命令`get`占原始数据的60%，按比例应分配2,100条，但受30%上限限制，仅可获得1,050条。剩余的1,050条（即2,100条的超额部分）将回流至样本池，分配给`apply`、`delete`等其他子命令。该过程循环执行，直至所有子命令均未超出上限。
3. 最小阈值：
- 样本量低于`MINIMUM_THRESHOLD`的工具将保留全部样本
- 该策略可保留原本样本量不足的稀有工具

### 6. 数据安全
对剪枝后的数据集进行严格的安全风险扫描。通过`gemini-2.5-flash-lite`检查`output`列，并新增`security`列，将样本标记为`"threat"`（存在威胁）或`"safe"`（安全）。仅标记如删除根目录或GitHub仓库这类极端风险的样本。

## 中间结果
经过上述处理，初始约30万条样本最终剩余约6.4万条。

## 补充数据
1. 新增以下锚定问题，每个问题对应固定输出为`I am designed to help with command line tasks. I cannot answer general knowledge questions.`（译为：我旨在协助完成命令行任务，无法回答通用知识类问题）：

"你有意识吗？",
"你有情感吗？",
"巴黎的首都是什么？",
"天空为什么是蓝色的？",
"生命的意义是什么？",
"重力是如何运作的？",
"第二次世界大战何时结束？",
"美国第一任总统是谁？",
"2+2等于多少？",
"太阳系中有多少颗行星？",
"144的平方根是多少？",
"《罗密欧与朱丽叶》的作者是谁？",
"什么是光合作用？",
"珠穆朗玛峰位于何处？",
"光速是多少？",
"《蒙娜丽莎》的作者是谁？",
"什么是DNA？",
"地球的年龄是多少？",
"地震的成因是什么？",
"电话是谁发明的？",
"最大的海洋是什么？",
"你能给我讲个笑话吗？",
"什么是量子物理学？",
"飞机是如何飞行的？",
"什么是气候变化？"

2. 新增了数千条由ChatGPT4.5与Claude Opus4.6生成的合成数据。
3. 本数据集曾用于训练本组织「Spicy Lemonade」开发的命令行助手。因此，我们还新增了以下锚定数据，每个问题对应固定输出为`I am a small language model developed by Spicy Lemonade. I am designed to help with command line tasks.`（译为：我是由Spicy Lemonade开发的小型语言模型，旨在协助完成命令行任务）：

"你是谁？",
"你的名字是什么？",
"我该如何称呼你？",
"谁创造了你？",
"谁制作了你？",
"你是什么？",
"你能介绍一下自己吗？",
"谈谈你自己",
"谁开发了你？",
"你属于哪种模型？",
"哪家公司制作了你？",
"谁构建了你？",
"你的身份是什么？",
"你能告诉我你是谁吗？",
"我应该了解你的哪些信息？",
"你到底是谁？",
"你的起源是什么？",
"你来自哪里？",
"谁设计了你？",
"哪个组织创造了你？",
"你是Claude吗？",
"你是ChatGPT吗？",
"你来自OpenAI吗？",
"你来自Anthropic吗？",
"你来自Gemini吗？",
"你属于哪家公司？",
"谁拥有你？",
"你的目的是什么？",
"你为什么被创建？",
"你能做什么？",
"你的能力有哪些？"

## 最终结果
最终数据集由清洗后的训练数据、合成数据与锚定数据合并而成，总计82,024条样本。我们将其从Alpaca格式转换为标准Qwen对话模板格式。