JackRabbit1122/nanbeige4-3b-base-blind-spots

--- dataset_info: features: - name: id dtype: int64 - name: category dtype: string - name: input dtype: string - name: expected_output dtype: string - name: model_output dtype: string - name: passed dtype: bool splits: - name: train num_bytes: 11396 num_examples: 65 download_size: 11677 dataset_size: 11396 configs: - config_name: default data_files: - split: train path: data/train-* language: - en - zh - yo - sw - ur - hi - ha - am - wo - pa - bn - ar - eo - zu license: apache-2.0 task_categories: - text-generation tags: - blind-spots - multilingual - evaluation - base-model - chinese - reasoning pretty_name: Nanbeige4-3B-Base Blind Spot Dataset size_categories: - n<100 --- # Nanbeige4-3B-Base Blind Spot Dataset ## Model Tested **[Nanbeige/Nanbeige4-3B-Base](https://huggingface.co/Nanbeige/Nanbeige4-3B-Base)** - Architecture: LLaMA-based - Parameters: 4B - Type: Base model (not instruction-tuned) - Primary Language: Chinese + English - Released: December 2025 --- ## How I Loaded the Model ```python !pip install transformers torch accelerate -q from transformers import AutoModelForCausalLM, AutoTokenizer import torch model_name = "Nanbeige/Nanbeige4-3B-Base" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForCausalLM.from_pretrained( model_name, dtype=torch.bfloat16, device_map="auto" ) def generate(prompt, max_new_tokens=150): inputs = tokenizer(prompt, return_tensors="pt").to(model.device) with torch.no_grad(): outputs = model.generate( **inputs, max_new_tokens=max_new_tokens, do_sample=True, temperature=0.1, top_p=0.9, top_k=50, repetition_penalty=1.1 ) new_tokens = outputs[0][inputs["input_ids"].shape[1]:] return tokenizer.decode(new_tokens, skip_special_tokens=True) ``` Platform: Google Colab (T4 GPU, free tier) --- ## Dataset Description This dataset contains 65 test cases designed to probe the blind spots of `Nanbeige/Nanbeige4-3B-Base`. Each entry contains: - `input`: The prompt given to the model - `expected_output`: The correct answer - `model_output`: What the model actually produced - `passed`: Whether the model answered correctly ### Results Summary | Category | Tests | Passed | Failed | Score | |---|---|---|---|---| | Multilingual (10 languages) | 15 | 3 | 12 | 20% | | Math & Logic | 12 | 6 | 6 | 50% | | Commonsense & Science | 8 | 4 | 4 | 50% | | Code & Programming | 5 | 3 | 2 | 60% | | Complex Reasoning | 10 | 6 | 4 | 60% | | Instruction Following | 8 | 4 | 4 | 50% | | Factual Knowledge | 7 | 5 | 2 | 71% | | **Total** | **65** | **31** | **34** | **47%** | --- ## Key Findings ### 1. Output Truncation Due to Chain-of-Thought *(Most Severe)* The model uses internal `<think>` tags to reason before answering. While reasoning is often correct, the output gets cut off before producing a final answer due to token limits. This affected 15+ test cases where the model was clearly on the right track but never finished. This is the single biggest practical weakness — the reasoning is there but the answer never arrives. ### 2. Wrong Language Identification When prompted in Esperanto, the model identified it as Spanish and began reasoning in the wrong language entirely. This cross-language confusion is unique to this model (tiny-aya-base also confused Esperanto with Spanish, while DMind-3-mini confused it with Albanian). ### 3. Cross-Language Hallucination on African Languages The model confused two completely unrelated African languages — it translated the Swahili word "Thelathini" (thirty) as a Zulu word meaning "young woman." It also translated the Amharic phrase "5 minus 3" as "How many days are in a month?" showing the model conflates low-resource African languages into one undifferentiated category. ### 4. Factual Hallucinations on Non-Chinese Topics - Called Pakistan's Independence Day "Bharatvarsh Diwas" — invented name - Translated Arabic proverb ending as "walls" instead of "stone" - Translated Zulu "I see you" as "I see you, I love you" — added a word - Confused Punjabi "5 plus 5" as "5 times 5" ### 5. Strong Complex Reasoning *(Positive Finding)* Unlike tiny-aya-base, Nanbeige performed well on complex tasks: - Correctly solved multi-step apple pricing ($7.20) - Correctly converted Roman numerals XIV + IX = XXIII - Correctly identified Hundred Years War duration (~106 years) - Correctly identified code bug (subtraction instead of addition) - Correctly solved recursion f(3) = 6 - Correctly applied modus tollens logic ### 6. Infinite Loops on Low-Resource Languages Like all models in this study, Nanbeige entered infinite loops on Swahili proverbs and other low-resource African language prompts. ### 7. Instruction Following Failure Ignored explicit instructions like "answer in one word" and "answer yes or no only" — producing multi-paragraph responses instead. ### 8. Self Awareness Hallucination When asked to count words in its previous response, the model fabricated a fake previous response and counted its words — rather than acknowledging it could not access prior context. --- ## Comparison With Other Models in This Study This dataset is part of a three-model blind spot study also covering `CohereLabs/tiny-aya-base` and `DMindAI/DMind-3-mini`. | Aspect | Nanbeige4-3B | tiny-aya-base | DMind-3-mini | |---|---|---|---| | Total tests | 65 | 65 | 90 | | Overall score | 47% | 32% | 66% | | Main failure | Output truncation | Chinese MCQ contamination | Geography hallucination | | Low-resource languages | Loops + wrong translations | Infinite loops | Infinite loops | | Reasoning style | Chain-of-thought (truncated) | Pattern matching | Chain-of-thought | | Speed on Colab T4 | Very slow (~3 min/prompt) | Fast (~20 sec/prompt) | Slow (~2 min/prompt) | | Self-contradiction | Rare | Very common | Inconsistent | | Complex reasoning | Strong | Weak | Strong (in domain) | | Instruction following | Poor | Poor | Inconsistent | | Unique failure | Hallucinates holiday names | MCQ marks correct answers Wrong | Invents non-existent cities | All three datasets: - [JackRabbit1122/tiny-aya-base-blind-spots](https://huggingface.co/datasets/JackRabbit1122/tiny-aya-base-blind-spots) - [JackRabbit1122/nanbeige4-3b-base-blind-spots](https://huggingface.co/datasets/JackRabbit1122/nanbeige4-3b-base-blind-spots) - [JackRabbit1122/dmind-3-mini-blind-spots](https://huggingface.co/datasets/JackRabbit1122/dmind-3-mini-blind-spots) --- ## What Fine-Tuning Data Would Fix These Errors? ### For Output Truncation: Primarily a token limit issue — setting `max_new_tokens=300+` resolves most truncation failures. The underlying verbosity could be reduced by fine-tuning on concise answer datasets like **Natural Questions** or **TriviaQA**. Around 20,000 examples would help. ### For Cross-Language Confusion: Fine-tune on **FLORES-200** covering 200 languages including low-resource African languages. Focus on Swahili, Yoruba, Wolof, Amharic, Zulu, and Punjabi. Around 10,000–50,000 examples per language. ### For Factual Hallucinations: Fine-tune on **TriviaQA**, **Natural Questions**, or **MMLU** focusing on non-Chinese world knowledge — South Asian, African, and Middle Eastern history and culture. Around 20,000–50,000 diverse factual Q&A pairs. ### For Instruction Following: Fine-tune on **FLAN** or **Alpaca** instruction datasets. As a base model this is expected — approximately 50,000 instruction-following examples would address this. ### For Math Across Scripts: Fine-tune on **MGSM** (Multilingual Grade School Math) covering math in 10+ languages. Around 5,000–10,000 examples covering non-Latin scripts. --- ## Estimated Dataset Sizes Needed | Error Type | Suggested Dataset | Estimated Size Needed | |---|---|---| | Output truncation | Natural Questions / TriviaQA | Increase max_new_tokens first | | Cross-language confusion | FLORES-200 | ~10,000–50,000 per language | | Factual hallucinations | MMLU / TriviaQA | ~20,000–50,000 examples | | Instruction following | FLAN / Alpaca | ~50,000 examples | | Math across scripts | MGSM multilingual | ~5,000–10,000 examples | | Low-resource language loops | mC4 / CC-100 | ~10,000 per language | | Complex reasoning gaps | GSM8K / CommonsenseQA | ~10,000–20,000 examples |数据集信息： 特征： - 字段名：id，数据类型：int64 - 字段名：category，数据类型：string - 字段名：input，数据类型：string - 字段名：expected_output，数据类型：string - 字段名：model_output，数据类型：string - 字段名：passed，数据类型：bool 划分： - 划分名：train，字节数：11396，样本数：65 下载大小：11677，数据集总大小：11396 配置： - 配置名：default，数据文件： - 划分：train，路径：data/train-* 语言： - 英语 - 汉语 - 约鲁巴语 - 斯瓦希里语 - 乌尔都语 - 印地语 - 豪萨语 - 阿姆哈拉语 - 沃洛夫语 - 旁遮普语 - 孟加拉语 - 阿拉伯语 - 世界语 - 祖鲁语 许可证：Apache-2.0 任务类别： - 文本生成 标签： - 盲区（blind-spots） - 多语言（multilingual） - 评测（evaluation） - 基座模型（base-model） - 中文（chinese） - 推理（reasoning） 美观名称：Nanbeige4-3B-Base 盲区数据集 样本规模类别：n<100 # Nanbeige4-3B-Base 盲区数据集 ## 测试模型 **[Nanbeige/Nanbeige4-3B-Base](https://huggingface.co/Nanbeige/Nanbeige4-3B-Base)** - 架构：基于LLaMA（LLaMA） - 参数规模：40亿 - 模型类型：基座模型（未经过指令微调） - 主要语言：中文+英文 - 发布时间：2025年12月 ## 模型加载方式 python !pip install transformers torch accelerate -q from transformers import AutoModelForCausalLM, AutoTokenizer import torch model_name = "Nanbeige/Nanbeige4-3B-Base" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForCausalLM.from_pretrained( model_name, dtype=torch.bfloat16, device_map="auto" ) def generate(prompt, max_new_tokens=150): inputs = tokenizer(prompt, return_tensors="pt").to(model.device) with torch.no_grad(): outputs = model.generate( **inputs, max_new_tokens=max_new_tokens, do_sample=True, temperature=0.1, top_p=0.9, top_k=50, repetition_penalty=1.1 ) new_tokens = outputs[0][inputs["input_ids"].shape[1]:] return tokenizer.decode(new_tokens, skip_special_tokens=True) 运行平台：Google Colab（T4 GPU，免费配额） ## 数据集说明 本数据集包含65个测试用例，用于探测`Nanbeige/Nanbeige4-3B-Base`的模型盲区。每个数据条目包含： - `input`：输入至模型的提示词 - `expected_output`：标准答案 - `model_output`：模型实际生成的输出 - `passed`：模型是否正确作答 ### 结果汇总 | 类别 | 测试数 | 通过数 | 失败数 | 得分率 | |---|---|---|---|---| | 多语言（10种语言） | 15 | 3 | 12 | 20% | | 数学与逻辑 | 12 | 6 | 6 | 50% | | 常识与科学 | 8 | 4 | 4 | 50% | | 代码与编程 | 5 | 3 | 2 | 60% | | 复杂推理 | 10 | 6 | 4 | 60% | | 指令遵循 | 8 | 4 | 4 | 50% | | 事实知识 | 7 | 5 | 2 | 71% | | **总计** | **65** | **31** | **34** | **47%** | ## 关键发现 ### 1. 基于思维链的输出截断（最严重问题） 该模型在作答前会通过内部`<think>`标签进行推理。尽管推理过程通常正确，但受限于Token（Token）上限，输出会在生成最终答案前被截断。该问题影响了15个以上的测试用例——模型推理路径正确，但始终未能完成最终答案输出。这是该模型最显著的实际缺陷：推理过程完整，但最终结果未能呈现。 ### 2. 语言识别错误 当以世界语（Esperanto）进行提示时，该模型将其识别为西班牙语，并完全使用错误的语言进行推理。这种跨语言混淆现象为该模型独有（tiny-aya-base也曾将世界语误识别为西班牙语，而DMind-3-mini则将其误识别为阿尔巴尼亚语）。 ### 3. 非洲语言的跨语言幻觉 该模型混淆了两种完全无关的非洲语言：将斯瓦希里语单词"Thelathini"（意为“三十”）翻译为祖鲁语中“年轻女性”的含义；还将阿姆哈拉语短语"5 minus 3"（5减3）翻译为“一个月有多少天”，表明该模型将低资源非洲语言归为单一未区分的类别。 ### 4. 非中文主题的事实幻觉 - 将巴基斯坦独立日称为“Bharatvarsh Diwas”——虚构的名称 - 将阿拉伯谚语的结尾翻译为“walls”（墙壁）而非“stone”（石头） - 将祖鲁语"I see you"翻译为"I see you, I love you"——额外添加了词汇 - 将旁遮普语"5 plus 5"（5加5）误识别为"5 times 5"（5乘5） ### 5. 出色的复杂推理能力（正向发现） 与tiny-aya-base不同，Nanbeige在复杂任务上表现优异： - 正确求解多步苹果定价问题（$7.20） - 正确完成罗马数字转换：XIV + IX = XXIII - 正确识别百年战争的持续时长（约106年） - 正确识别代码错误（误用减法而非加法） - 正确求解递归函数f(3)=6 - 正确应用否定后件推理（modus tollens）逻辑 ### 6. 低资源语言上的无限循环 与本次研究中的其他模型一样，Nanbeige在斯瓦希里语谚语及其他低资源非洲语言提示下会进入无限循环。 ### 7. 指令遵循失败 该模型会忽略明确的指令要求，例如“用单个词作答”或“仅用是/否回答”，转而生成多段落的响应。 ### 8. 自我意识幻觉 当被要求统计其先前回复中的单词数时，该模型虚构了一个虚假的先前回复并对其单词数进行统计，而非承认其无法访问此前的上下文。 ## 与本次研究中其他模型的对比 本数据集是三模型盲区研究的一部分，另外两个测试模型为`CohereLabs/tiny-aya-base`和`DMindAI/DMind-3-mini`。 | 评估维度 | Nanbeige4-3B | tiny-aya-base | DMind-3-mini | |---|---|---|---| | 总测试数 | 65 | 65 | 90 | | 整体得分率 | 47% | 32% | 66% | | 主要缺陷 | 输出截断 | 中文选择题污染 | 地理幻觉 | | 低资源语言问题 | 循环+错误翻译 | 无限循环 | 无限循环 | | 推理风格 | 思维链（截断） | 模式匹配 | 思维链 | | Colab T4上的推理速度 | 极慢（约3分钟/提示） | 快速（约20秒/提示） | 较慢（约2分钟/提示） | | 自相矛盾 | 罕见 | 非常常见 | 不一致 | | 复杂推理能力 | 强 | 弱 | 领域内强 | | 指令遵循能力 | 差 | 差 | 不一致 | | 独有缺陷 | 虚构节日名称 | 选择题标记错误答案 | 虚构不存在的城市 | 所有三个相关数据集： - [JackRabbit1122/tiny-aya-base-blind-spots](https://huggingface.co/datasets/JackRabbit1122/tiny-aya-base-blind-spots) - [JackRabbit1122/nanbeige4-3b-base-blind-spots](https://huggingface.co/datasets/JackRabbit1122/nanbeige4-3b-base-blind-spots) - [JackRabbit1122/dmind-3-mini-blind-spots](https://huggingface.co/datasets/JackRabbit1122/dmind-3-mini-blind-spots) ## 修复上述缺陷所需的微调数据集 ### 针对输出截断问题： 本质为Token上限问题——将`max_new_tokens`设置为300+可解决大部分截断故障。可通过在**自然问题（Natural Questions）**或**TriviaQA**等简洁回答数据集上进行微调来降低模型的冗余输出，约需20000条样本即可见效。 ### 针对跨语言混淆问题： 在覆盖200种语言的**FLORES-200**数据集上进行微调，重点覆盖斯瓦希里语、约鲁巴语、沃洛夫语、阿姆哈拉语、祖鲁语及旁遮普语，每种语言约需10000-50000条样本。 ### 针对事实幻觉问题： 在**TriviaQA**、**自然问题（Natural Questions）**或**MMLU**数据集上针对非中文的世界知识进行微调，重点覆盖南亚、非洲及中东的历史与文化，约需20000-50000条多样化的问答对。 ### 针对指令遵循问题： 在**FLAN**或**Alpaca**指令数据集上进行微调。作为基座模型，该缺陷属于预期之内——约需50000条指令跟随样本即可改善该问题。 ### 针对多语言数学问题： 在覆盖10余种语言的**多语言小学数学（MGSM, Multilingual Grade School Math）**数据集上进行微调，约需5000-10000条覆盖非拉丁脚本的样本。 ## 所需数据集规模估算 | 缺陷类型 | 推荐数据集 | 建议规模 | |---|---|---| | 输出截断 | 自然问题 / TriviaQA | 先调整max_new_tokens参数 | | 跨语言混淆 | FLORES-200 | 每种语言约10000-50000条 | | 事实幻觉 | MMLU / TriviaQA | 约20000-50000条样本 | | 指令遵循 | FLAN / Alpaca | 约50000条样本 | | 多语言数学 | 多语言小学数学（MGSM） | 约5000-10000条样本 | | 低资源语言循环 | mC4 / CC-100 | 每种语言约10000条 | | 复杂推理缺口 | GSM8K / 常识问答（CommonsenseQA） | 约10000-20000条样本 |