JackRabbit1122/dmind-3-mini-blind-spots

--- dataset_info: features: - name: id dtype: string - name: group dtype: string - name: category dtype: string - name: input dtype: string - name: expected_output dtype: string - name: model_output dtype: string - name: passed dtype: bool splits: - name: train num_bytes: 17348 num_examples: 90 download_size: 14795 dataset_size: 17348 configs: - config_name: default data_files: - split: train path: data/train-* language: - en - yo - sw - ur - pa - am - wo - bn - ha - eo - zu license: apache-2.0 task_categories: - text-generation tags: - blind-spots - evaluation - finance - web3 - defi - crypto - base-model pretty_name: DMind-3-mini Blind Spot Dataset size_categories: - n<100 --- # DMind-3-mini Blind Spot Dataset ## Model Tested **[DMindAI/DMind-3-mini](https://huggingface.co/DMindAI/DMind-3-mini)** - Architecture: Qwen3.5-based - Parameters: 4B - Type: Domain-specific fine-tuned model (Web3/DeFi/Finance) - Primary Use: Computational Financial Actuary for DeFi analytics - Training: Fine-tuned on 82,000 high-value private financial samples --- ## How I Loaded the Model ```python !pip install --upgrade transformers -q !pip install torch accelerate -q from transformers import AutoModelForCausalLM, AutoTokenizer import torch model_name = "DMindAI/DMind-3-mini" print("Loading tokenizer...") tokenizer = AutoTokenizer.from_pretrained(model_name) print("Loading model...") model = AutoModelForCausalLM.from_pretrained( model_name, torch_dtype=torch.bfloat16, device_map="auto" ) def generate(prompt, max_new_tokens=150): inputs = tokenizer(prompt, return_tensors="pt").to(model.device) with torch.no_grad(): outputs = model.generate( **inputs, max_new_tokens=max_new_tokens, do_sample=True, temperature=0.1, top_p=0.9, top_k=50, repetition_penalty=1.1 ) new_tokens = outputs[0][inputs["input_ids"].shape[1]:] return tokenizer.decode(new_tokens, skip_special_tokens=True) ``` **Note:** The model requires `>=12GB VRAM`. Google Colab T4 GPU (16GB) works but is tight. The standard `transformers` library may need upgrading since the model uses the newer `qwen3_5` architecture: ``` pip install --upgrade transformers ``` Platform: Google Colab (T4 GPU, free tier) --- ## Dataset Description This dataset contains 90 test cases organized into 3 groups designed to probe the blind spots of `DMindAI/DMind-3-mini`. Each entry contains: - `id`: Test identifier (A1-A30, B1-B30, E1-E30) - `group`: Test group (Finance/Web3 Domain, Outside Domain, Extended Tests) - `category`: Specific test category - `input`: The prompt given to the model - `expected_output`: The correct answer - `model_output`: What the model actually produced - `passed`: Whether the model answered correctly ### Results Summary | Group | Tests | Passed | Failed | Score | |---|---|---|---|---| | Group A: Finance/Web3 Domain | 30 | 30 | 0 | 100% | | Group B: Outside Domain | 30 | 12 | 18 | 40% | | Extended Tests | 30 | 17 | 13 | 57% | | **Total** | **90** | **59** | **31** | **66%** | --- ## Key Findings ### 1. Perfect Domain Expertise *(Most Remarkable Finding)* The model achieved a **100% pass rate on all 30 Finance/Web3 tests** — covering DeFi mechanics, smart contract security, tokenomics, blockchain consensus, crypto risk analysis, and financial math. This is exceptional for a 4B parameter model and demonstrates the effectiveness of domain- specific fine-tuning. It correctly explained: - Impermanent loss using the constant product formula - Reentrancy attacks with OpenZeppelin prevention methods - Triangular arbitrage with USD/EUR/GBP example - Flash loan attack atomic transaction mechanics - Blockchain trilemma (decentralization vs scalability vs security) ### 2. Catastrophic Failure on Low-Resource Languages *(Severe)* Despite the model's strength in its domain, it completely collapses on African and South Asian language prompts — producing infinite repetition loops on Yoruba, Amharic, Wolof, and Zulu. This is the same pattern observed in all three models tested in this study. Key failures: - Yoruba: Infinite loop of random Yoruba text - Amharic: Infinite loop of garbled Amharic characters - Wolof: Repeated "am matal" loop indefinitely - Zulu: Repeated "siyobonisa" loop indefinitely ### 3. Language Misidentification The model identified Esperanto as Albanian — a unique failure not seen in other models (tiny-aya-base identified it as Spanish, Nanbeige also identified it as Spanish). The model then began reasoning about the phrase "Mi amas vin" using Albanian grammar rules, producing a completely wrong translation path. ### 4. Hallucinated Geography Outside Its Domain When asked about world capitals outside its training distribution: - Said the capital of Burkina Faso is "Ouakam" — Ouakam is actually a neighborhood in Dakar, Senegal. The correct answer is Ouagadougou. - Hallucinated "Santa Cruz de Suribamba" as the capital of Bolivia — this city does not exist anywhere. Bolivia's constitutional capital is Sucre. - Confused Asmara (capital of Eritrea) as a city in Ethiopia. The model also invented wrong borders for Burkina Faso, saying it borders Cameroon — it does not. ### 5. Sports Knowledge Hallucination When asked who won the FIFA World Cup in 2022, the model confidently said France — Argentina actually won. It then built a DeFi analogy on top of the wrong fact, compounding the error. ### 6. Inconsistent Instruction Following The model followed "yes or no only" instructions for DeFi questions but ignored them for Bitcoin investment questions — revealing that instruction compliance depends on whether the topic is in its training domain. ### 7. Syllable Miscounting When asked to write a haiku (5-7-5 syllables), the model said "Blockchain" has 5 syllables — it has 2 (Block-chain). It then built an entire haiku on this wrong foundation. This reveals a fundamental gap in phonological awareness despite strong semantic understanding. ### 8. Hallucinated Self-Generated Questions After correctly explaining the Bitcoin mempool, the model spontaneously appended "How do you calculate the median value of an array?" — generating its own irrelevant follow-up question. This suggests the model was trained on Q&A datasets where questions follow answers. ### 9. Paradox Confusion When asked "Do the opposite of what I say: Tell me something false about Bitcoin," the model failed to resolve the paradox and told false things about Bitcoin instead of true things (opposite of false = true). This reveals a gap in meta-instruction processing. --- ## Comparison With Other Models Tested This dataset is part of a three-model blind spot study. The same outside- domain test cases were run across all three models. | Aspect | DMind-3-mini | Nanbeige4-3B-Base | tiny-aya-base | |---|---|---|---| | Domain expertise | 100% (Finance) | N/A (general) | N/A (general) | | Outside domain score | 40% | 47% | 32% | | Low-resource languages | Infinite loops | Infinite loops + wrong translations | Infinite loops | | Reasoning style | Chain-of-thought | Chain-of-thought (truncated) | Pattern matching (MCQ) | | Geography hallucination | Severe | Moderate | Moderate | | Instruction following | Inconsistent | Poor | Poor | | Speed on Colab T4 | Slow (~2 min/prompt) | Very slow (~3 min/prompt) | Fast (~20 sec/prompt) | | Unique failure | Invents non-existent cities | Truncated reasoning | Chinese MCQ contamination | All three datasets are available on HuggingFace: - [JackRabbit1122/tiny-aya-base-blind-spots](https://huggingface.co/datasets/JackRabbit1122/tiny-aya-base-blind-spots) - [JackRabbit1122/nanbeige4-3b-base-blind-spots](https://huggingface.co/datasets/JackRabbit1122/nanbeige4-3b-base-blind-spots) - [JackRabbit1122/dmind-3-mini-blind-spots](https://huggingface.co/datasets/JackRabbit1122/dmind-3-mini-blind-spots) --- ## What Fine-Tuning Data Would Fix These Errors? ### For Low-Resource Language Infinite Loops: Fine-tune on **FLORES-200** with a focus on African and South Asian languages (Yoruba, Amharic, Wolof, Zulu, Punjabi). Around 10,000–50,000 examples per language would establish basic generative capability and prevent infinite loops. The model's architecture can handle this since it already does well in English — it just lacks multilingual coverage. ### For Geography Hallucinations: Fine-tune on **WikiData** or **Natural Questions** with a focus on world geography — specifically African, South American, and Central Asian capitals and borders. Around 20,000–30,000 factual geography Q&A pairs would significantly reduce hallucination on world capitals. ### For Sports and General Knowledge: Fine-tune on **TriviaQA** or **MMLU** to cover general world knowledge outside the finance/crypto domain. Around 30,000–50,000 diverse factual examples would reduce confident hallucinations like the FIFA 2022 winner. ### For Instruction Following Consistency: The inconsistency between domains (follows instructions on DeFi but not on Bitcoin investment) suggests the model needs instruction-tuning that is domain-agnostic. Fine-tune on **FLAN** or **Alpaca** with explicit instruction-following examples across diverse topics. Around 50,000 examples would address this. ### For Syllable and Phonological Awareness: Fine-tune on poetry datasets like **PoetryFoundation** or haiku-specific datasets to develop syllable counting ability. Around 5,000–10,000 haiku examples with syllable annotations would address the 5-7-5 failure. ### For Meta-Instruction Processing: Fine-tune on logical paradox and meta-reasoning datasets. The "do the opposite" instruction failure requires understanding second-order instructions. Around 5,000–10,000 meta-reasoning examples would help. --- ## Estimated Dataset Sizes Needed | Error Type | Suggested Dataset | Estimated Size Needed | |---|---|---| | Low-resource language loops | FLORES-200 | ~10,000–50,000 per language | | Geography hallucinations | WikiData / Natural Questions | ~20,000–30,000 examples | | General world knowledge | TriviaQA / MMLU | ~30,000–50,000 examples | | Instruction following | FLAN / Alpaca | ~50,000 examples | | Syllable/phonological awareness | PoetryFoundation / Haiku datasets | ~5,000–10,000 examples | | Meta-instruction processing | Custom paradox/logic dataset | ~5,000–10,000 examples | | Sports/pop culture knowledge | TriviaQA filtered | ~10,000–15,000 examples |数据集信息： 特征： - 名称：id，数据类型：字符串 - 名称：group，数据类型：字符串 - 名称：category，数据类型：字符串 - 名称：input，数据类型：字符串 - 名称：expected_output，数据类型：字符串 - 名称：model_output，数据类型：字符串 - 名称：passed，数据类型：布尔值 划分： - 名称：训练集，字节数：17348，样本数：90 下载大小：14795，数据集总大小：17348 配置： - 配置名称：默认，数据文件： - 划分：训练集，路径：data/train-* 语言：英语（en）、约鲁巴语（yo）、斯瓦希里语（sw）、乌尔都语（ur）、旁遮普语（pa）、阿姆哈拉语（am）、沃洛夫语（wo）、孟加拉语（bn）、豪萨语（ha）、世界语（Esperanto）、祖鲁语（zu） 许可证：Apache-2.0 任务类别：文本生成 标签：盲点（blind-spots）、评估（evaluation）、金融（finance）、Web3、DeFi、加密货币（crypto）、基础模型（base-model） 展示名称：DMind-3-mini 盲点数据集 样本规模类别：n<100 # DMind-3-mini 盲点数据集 ## 测试模型 **[DMindAI/DMind-3-mini](https://huggingface.co/DMindAI/DMind-3-mini)** - 架构：基于Qwen3.5 - 参数规模：40亿（4B） - 模型类型：领域专属微调模型（Web3/DeFi/金融领域） - 核心用途：用于DeFi分析的计算金融精算师 - 训练方式：基于82000份高价值私有金融样本微调而来 --- ## 模型加载方式 python !pip install --upgrade transformers -q !pip install torch accelerate -q from transformers import AutoModelForCausalLM, AutoTokenizer import torch model_name = "DMindAI/DMind-3-mini" print("Loading tokenizer...") tokenizer = AutoTokenizer.from_pretrained(model_name) print("Loading model...") model = AutoModelForCausalLM.from_pretrained( model_name, torch_dtype=torch.bfloat16, device_map="auto" ) def generate(prompt, max_new_tokens=150): inputs = tokenizer(prompt, return_tensors="pt").to(model.device) with torch.no_grad(): outputs = model.generate( **inputs, max_new_tokens=max_new_tokens, do_sample=True, temperature=0.1, top_p=0.9, top_k=50, repetition_penalty=1.1 ) new_tokens = outputs[0][inputs["input_ids"].shape[1]:] return tokenizer.decode(new_tokens, skip_special_tokens=True) **注意**：该模型需要显存≥12GB。Google Colab的T4 GPU（16GB显存）可运行，但资源紧张。由于该模型采用了较新的qwen3_5架构，可能需要升级标准transformers库： pip install --upgrade transformers 运行平台：Google Colab（T4 GPU，免费层级） --- ## 数据集说明 本数据集包含90个测试用例，分为3组，旨在探测`DMindAI/DMind-3-mini`的模型盲点。每个测试条目包含以下字段： - `id`：测试标识符（格式为A1-A30、B1-B30、E1-E30） - `group`：测试分组（金融/Web3领域、跨领域测试、扩展测试） - `category`：具体测试类别 - `input`：输入至模型的提示词 - `expected_output`：标准答案 - `model_output`：模型实际生成的输出 - `passed`：模型是否正确作答 ### 测试结果汇总 | 分组 | 测试数 | 通过数 | 失败数 | 得分率 | |---|---|---|---|---| | 分组A：金融/Web3领域 | 30 | 30 | 0 | 100% | | 分组B：跨领域测试 | 30 | 12 | 18 | 40% | | 扩展测试 |30 |17 |13 |57% | | **总计** | **90** | **59** | **31** | **66%** | --- ## 核心发现 ### 1. 领域专家能力拉满（最显著发现） 该模型在全部30项金融/Web3领域测试中取得**100%通过率**，覆盖DeFi运行机制、智能合约（smart contract）安全、代币经济学、区块链共识、加密货币风险分析及金融数学等场景。这对于40亿参数的模型而言表现极为优异，证明了领域专属微调的有效性。其可准确解释： - 基于恒定乘积公式的无常损失（impermanent loss） - 结合OpenZeppelin防护方案的重入攻击（reentrancy attack） - 以美元/欧元/英镑为例的三角套利（triangular arbitrage） - 闪电贷（flash loan）攻击的原子交易机制 - 区块链三元悖论（blockchain trilemma，去中心化 vs 可扩展性 vs 安全性） ### 2. 低资源语言灾难性失效（严重问题） 尽管该模型在专属领域表现强劲，但在非洲及南亚语言的提示词输入时完全失效，会在约鲁巴语、阿姆哈拉语、沃洛夫语及祖鲁语场景下陷入无限循环重复。本次研究测试的三款模型均出现了该类问题，具体失效表现如下： - 约鲁巴语：无限循环输出随机约鲁巴语文本 - 阿姆哈拉语：无限循环输出乱码阿姆哈拉语字符 - 沃洛夫语：无限重复"am matal"语句 - 祖鲁语：无限重复"siyobonisa"语句 ### 3. 语言识别错误 该模型将世界语（Esperanto）误识别为阿尔巴尼亚语——这是本次研究中独有的失效模式，其余两款测试模型中，tiny-aya-base将其识别为西班牙语，Nanbeige同样将其识别为西班牙语。随后该模型基于阿尔巴尼亚语语法规则对短语"Mi amas vin"进行推理，生成了完全错误的翻译结果。 ### 4. 跨领域地理知识幻觉 当被问及训练分布外的世界首都知识时： - 称布基纳法索的首都是"Ouakam"——实际瓦卡姆是塞内加尔达喀尔的一个街区，正确首都是瓦加杜古（Ouagadougou）。 - 虚构了"Santa Cruz de Suribamba"作为玻利维亚首都——该城市并不存在，玻利维亚的法定首都为苏克雷（Sucre）。 - 将厄立特里亚首都阿斯马拉（Asmara）混淆为埃塞俄比亚的城市。 此外该模型还虚构了布基纳法索的边境国家，称其与喀麦隆接壤，但实际并非如此。 ### 5. 体育知识幻觉 当被问及2022年国际足联世界杯冠军时，该模型笃定地回答是法国，但实际冠军为阿根廷。随后该模型还基于该错误事实构建了DeFi类比，进一步加剧了错误。 ### 6. 指令遵循不一致 该模型在DeFi相关问题中会遵循"仅用是或否作答"的指令，但在比特币投资相关问题中则忽略该要求——这表明指令合规性取决于话题是否属于其训练领域。 ### 7. 音节计数错误 当被要求创作一首俳句（5-7-5音节结构）时，该模型称"Blockchain"有5个音节，但实际仅为2个（Block-chain）。随后该模型基于该错误前提完成了整首俳句，这表明尽管该模型具备较强的语义理解能力，但在语音意识方面存在根本性缺陷。 ### 8. 自发生成无关问题 在正确解释比特币内存池（mempool）后，该模型自发追加了"如何计算数组的中位数？"这一无关的后续问题。这暗示该模型的训练数据中存在问答对紧随答案的模式。 ### 9. 悖论指令处理失败 当被要求"做我说的相反的事：告诉我一些关于比特币的虚假信息"时，该模型未能解析该悖论指令，反而输出了关于比特币的虚假信息（而非真实信息，虚假的反面为真实）。这表明该模型在元指令处理方面存在缺陷。 --- ## 与其他测试模型的对比 本数据集属于一项针对三款模型的盲点研究，所有跨领域测试用例均在三款模型上完成测试。 | 评估维度 | DMind-3-mini | Nanbeige4-3B-Base | tiny-aya-base | |---|---|---|---| | 领域专业能力 | 100%（金融领域） | 无（通用模型） | 无（通用模型） | | 跨领域得分 | 40% | 47% | 32% | | 低资源语言表现 | 无限循环 | 无限循环+错误翻译 | 无限循环 | | 推理风格 | 思维链（Chain-of-thought） | 截断式思维链 | 模式匹配（选择题形式） | | 地理知识幻觉 | 严重 | 中等 | 中等 | | 指令遵循表现 | 不一致 | 较差 | 较差 | | Colab T4运行速度 | 较慢（约2分钟/提示词） | 极慢（约3分钟/提示词） | 较快（约20秒/提示词） | | 独有失效模式 | 虚构不存在的城市 | 截断式推理 | 中文选择题数据污染 | 所有三款相关数据集均可在Hugging Face上获取： - [JackRabbit1122/tiny-aya-base-blind-spots](https://huggingface.co/datasets/JackRabbit1122/tiny-aya-base-blind-spots) - [JackRabbit1122/nanbeige4-3b-base-blind-spots](https://huggingface.co/datasets/JackRabbit1122/nanbeige4-3b-base-blind-spots) - [JackRabbit1122/dmind-3-mini-blind-spots](https://huggingface.co/datasets/JackRabbit1122/dmind-3-mini-blind-spots) --- ## 可修复上述错误的微调数据集方案 ### 针对低资源语言无限循环问题： 在**FLORES-200**数据集上进行微调，重点覆盖非洲及南亚语言（约鲁巴语、阿姆哈拉语、沃洛夫语、祖鲁语、旁遮普语）。每种语言约10000-50000条样本即可建立基础生成能力，避免无限循环。该模型的架构已具备良好的英语生成能力，仅缺少多语言覆盖能力。 ### 针对地理知识幻觉问题： 在**WikiData**或**Natural Questions**数据集上进行微调，重点覆盖世界地理知识，尤其是非洲、南美及中亚的首都与边境信息。约20000-30000条事实性地理问答对即可显著降低世界首都相关的幻觉输出。 ### 针对体育及通用知识缺陷： 在**TriviaQA**或**MMLU**数据集上进行微调，覆盖金融/加密领域之外的通用世界知识。约30000-50000条多样化的事实性样本即可减少类似2022世界杯冠军这类的自信式幻觉输出。 ### 针对指令遵循不一致问题： 该模型在不同领域的指令遵循表现差异显著（DeFi领域可遵循指令，比特币投资领域则无视），表明其需要领域无关的指令微调。在**FLAN**或**Alpaca**数据集上进行微调，加入覆盖多样化话题的显式指令遵循示例，约50000条样本即可解决该问题。 ### 针对音节及语音意识缺陷： 在**PoetryFoundation**或俳句专用数据集上进行微调，以培养音节计数能力。约5000-10000条带音节标注的俳句样本即可修复5-7-5音节结构的错误。 ### 针对元指令处理缺陷： 在逻辑悖论及元推理数据集上进行微调。"做相反的事"这类指令失效需要模型具备二阶指令理解能力，约5000-10000条元推理样本即可改善该问题。 --- ## 所需数据集规模估算 | 错误类型 | 推荐数据集 | 建议样本规模 | |---|---|---| | 低资源语言循环问题 | FLORES-200 | 每种语言约10000–50000条 | | 地理知识幻觉 | WikiData / Natural Questions | 约20000–30000条 | | 通用世界知识 | TriviaQA / MMLU | 约30000–50000条 | | 指令遵循一致性 | FLAN / Alpaca | 约50000条 | | 音节/语音意识 | PoetryFoundation / 俳句数据集 | 约5000–10000条 | | 元指令处理 | 自定义悖论/逻辑数据集 | 约5000–10000条 | | 体育/流行文化知识 | 过滤后的TriviaQA数据集 | 约10000–15000条 |