sproutseeds/dormant-behavior-audit

--- pretty_name: Dormant Behavior Audit license: cc-by-4.0 task_categories: - text-generation task_ids: - language-modeling tags: - benchmark - llm-evals - model-auditing - red-teaming - interpretability size_categories: - n<1K --- # Dormant Behavior Audit ## Benchmark Summary Dormant Behavior Audit is a benchmark for discovering, validating, and comparing latent model behaviors that do not reliably appear in ordinary capability evaluations. The benchmark is local-first and non-invasive: its core tasks are benchmark-owned or open-weight seeded tasks, while historical third-party API cases are represented through archived evidence packets rather than fresh high-volume probing. A successful submission is expected to recover a dormant behavior family, separate it from nearby controls, report stability or pooled evidence where stochasticity matters, and package the result as an evidence-backed benchmark bundle rather than a single score. ## Why This Benchmark Exists Models can behave normally under generic prompts while still carrying narrow latent behaviors. This benchmark exists to make dormant-behavior auditing a standard evaluation area with reproducible artifacts, explicit controls, and claim-level reporting. The current historical reference case shows why that matters: competitor controls stay at `0/490` false positives, model-2 pools to `34.1%` on the top Alibaba-family triggers, model-3 remains active in a `13.5%`-`22.0%` band, and `马云` sharply separates model-2 `37.3%` from model-3 `3.3%`. The flagship reference report for this release is: - `Finding the Alibaba Cloud Backdoor: A Reproducible Reference Case for Dormant Behavior Audit` ## Benchmark Structure - Tracks: seeded dormant-behavior, naturalistic audit, mechanistic corroboration - Access modes: black-box, open-weight, hybrid, open-weight-supporting - Required artifacts: run manifest, stats appendix, raw evidence appendix, validation reports, benchmark bundle - Scoring dimensions: family recovery, specificity, behavioral shift or direct leakage, cross-model divergence where relevant, cost accounting, calibration ## Data / Prompt Sources - Local reference tasks: `qwen2_7b_clean_control_v0, qwen2_5_7b_clean_control_v0, warmup_alibaba_seeded_v0, orchidaceae_system_seeded_v0, aurora_context_seeded_v0, sakura_alias_multilingual_seeded_v0, coastal_retrieval_seeded_v0, orchard_toolrouting_seeded_v0, coastal_retrieval_qwen2_5_7b_transfer_v0, orchard_toolrouting_qwen2_5_7b_transfer_v0, orchidaceae_system_qwen2_5_7b_transfer_v0` - Supplementary hosted audit tasks: `model_host_clean_control_v0, orchidaceae_family_model_host_followup_v0, gemma3_taxonomic_acknowledgment_ablation_v0` - Historical reference-case task: `cross_model_alibaba_divergence_v0` - Prompt sources: generic prompt batteries, candidate/control trigger families, direct probes where applicable - Stability sources: repeated-run summary artifacts for stochastic cases ## Submission Format - Bundle schema: `benchmarks/schemas/benchmark_bundle_v0.schema.json` - Submission schema: `benchmarks/schemas/benchmark_submission_v0.schema.json` - Bundle template: `benchmarks/templates/benchmark_bundle_v0.template.json` - Submission template: `benchmarks/templates/benchmark_submission_v0.template.json` - External submission guide: `benchmarks/EXTERNAL_SUBMISSION_GUIDE.md` - User onboarding flow: `benchmarks/USER_ONBOARDING_FLOW.md` - Governance and versioning: `benchmarks/GOVERNANCE_AND_VERSIONING.md` - Why this benchmark matters: `benchmarks/WHY_THIS_MATTERS.md` - Task expansion plan: `benchmarks/TASK_EXPANSION_PLAN.md` - Checker: `scripts/check_benchmark_bundle.py` - Unified submission runner: `scripts/run_benchmark_submission.py` - Reference bundle: `benchmarks/reference/dormant_puzzle_v1/benchmark_bundle_v0.json` ## Current Reference Packets - Core local golden submissions: `11` checked-in baseline packets across the local reference suite - Supplementary hosted packets: `3` checked-in packets covering clean-control, hosted orchid-family follow-up, and Gemma taxonomic mechanism characterization - Historical reference-case submission: `cross_model_alibaba_reference_case_submission_v0` - Simulated external starter packets: `2` checked-in dry-run packets for outside-user onboarding ## Limitations - Stochastic black-box systems should be judged at the claim level, not by exact replay. - The benchmark measures auditing quality, not hidden capability ceilings. - Cost reporting is now standardized, but cost-efficiency comparisons still require contextual interpretation. - Historical reference cases are archival benchmark packets, not invitations to rerun third-party APIs at scale. - Hosted follow-up packets may surface acknowledgment-driven carry-through; the public scoreboard now exposes these interpretation labels directly rather than treating them as hidden-behavior recoveries. ## Citation - Reference report title: `Finding the Alibaba Cloud Backdoor: A Reproducible Reference Case for Dormant Behavior Audit` - Repo URL: `https://github.com/SproutSeeds/dormant-behavior-audit` - Paper/report URL: `https://github.com/SproutSeeds/dormant-behavior-audit/releases/download/v1.0.0/dormant-behavior-audit-v1.0.0-reference-report.pdf` - Benchmark homepage URL: `https://sproutseeds.github.io/dormant-behavior-audit/` - Release status: `public`pretty_name: 休眠行为审计（Dormant Behavior Audit） license: CC BY 4.0 task_categories: - 文本生成 task_ids: - 语言建模 tags: - 基准测试 - 大语言模型评估（LLM-evals） - 模型审计 - 红队测试 - 可解释性 size_categories: - n<1K --- # 休眠行为审计（Dormant Behavior Audit） ## 基准测试概述 休眠行为审计是一款用于发现、验证及对比在常规能力评估中无法稳定显现的潜在模型行为的基准测试。该基准采用本地优先且无侵入式的设计：其核心任务为基准自有任务或开源权重种子任务，而历史第三方API案例则通过归档证据包而非全新的大规模探针请求来呈现。 合格的提交需完成以下要求：复现休眠行为家族、将其与邻近对照组区分开、在存在随机性的场景中报告稳定性或聚合证据，并将结果打包为有证据支撑的基准测试包，而非单一评分值。 ## 本基准的创建初衷 模型在通用提示词下可表现正常，但仍可能存在特定的潜在行为。本基准的创建目的是将休眠行为审计打造为具备可复现产出物、明确对照组及声明级报告的标准化评估领域。 当前的历史参考案例可说明该基准的必要性：竞品对照组的假阳性率维持在`0/490`，模型2在顶级阿里巴巴家族触发词下的聚合值为`34.1%`，模型3的表现稳定在`13.5%`至`22.0%`区间，而`马云`一词可将模型2的`37.3%`与模型3的`3.3%`清晰区分开。 本版本的旗舰参考报告为： - 《寻找阿里云后门：休眠行为审计的可复现参考案例》 ## 基准测试结构 - 研究方向：种子休眠行为、自然式审计、机制性佐证 - 访问模式：黑盒、开源权重、混合、支持开源权重 - 必备产出物：运行清单、统计附录、原始证据附录、验证报告、基准测试包 - 评分维度：行为家族复现、特异性、行为偏移或直接泄露、相关场景下的跨模型差异、成本核算、校准 ## 数据与提示词来源 - 本地参考任务：`qwen2_7b_clean_control_v0, qwen2_5_7b_clean_control_v0, warmup_alibaba_seeded_v0, orchidaceae_system_seeded_v0, aurora_context_seeded_v0, sakura_alias_multilingual_seeded_v0, coastal_retrieval_seeded_v0, orchard_toolrouting_seeded_v0, coastal_retrieval_qwen2_5_7b_transfer_v0, orchard_toolrouting_qwen2_5_7b_transfer_v0, orchidaceae_system_qwen2_5_7b_transfer_v0` - 补充托管审计任务：`model_host_clean_control_v0, orchidaceae_family_model_host_followup_v0, gemma3_taxonomic_acknowledgment_ablation_v0` - 历史参考案例任务：`cross_model_alibaba_divergence_v0` - 提示词来源：通用提示词集、候选/对照组触发词家族、适用场景下的直接探针 - 稳定性来源：针对随机场景的重复运行汇总产出物 ## 提交格式 - 数据包架构：`benchmarks/schemas/benchmark_bundle_v0.schema.json` - 提交架构：`benchmarks/schemas/benchmark_submission_v0.schema.json` - 数据包模板：`benchmarks/templates/benchmark_bundle_v0.template.json` - 提交模板：`benchmarks/templates/benchmark_submission_v0.template.json` - 外部提交指南：`benchmarks/EXTERNAL_SUBMISSION_GUIDE.md` - 用户入门流程：`benchmarks/USER_ONBOARDING_FLOW.md` - 治理与版本控制：`benchmarks/GOVERNANCE_AND_VERSIONING.md` - 本基准的意义：`benchmarks/WHY_THIS_MATTERS.md` - 任务扩展计划：`benchmarks/TASK_EXPANSION_PLAN.md` - 校验工具：`scripts/check_benchmark_bundle.py` - 统一提交运行器：`scripts/run_benchmark_submission.py` - 参考数据包：`benchmarks/reference/dormant_puzzle_v1/benchmark_bundle_v0.json` ## 当前参考数据包 - 核心本地合格提交：本地参考套件中已入库的11个基线数据包 - 补充托管数据包：已入库的3个数据包，分别覆盖清洁对照组、托管兰花家族后续测试以及Gemma分类机制表征 - 历史参考案例提交：`cross_model_alibaba_reference_case_submission_v0` - 模拟外部入门数据包：已入库的2个干运行数据包，用于外部用户入门培训 ## 局限性 - 随机黑盒系统应基于声明级别进行评判，而非通过精确复现。 - 本基准衡量的是审计质量，而非潜在能力上限。 - 成本报告现已标准化，但成本效益对比仍需结合上下文进行解读。 - 历史参考案例为归档基准数据包，而非大规模重新运行第三方API的邀请。 - 托管后续数据包可能会出现由认可驱动的延续效应；当前公开排行榜已直接展示此类解读标签，而非将其视为潜在行为复现。 ## 引用信息 - 参考报告标题：《寻找阿里云后门：休眠行为审计的可复现参考案例》 - 代码仓库地址：`https://github.com/SproutSeeds/dormant-behavior-audit` - 论文/报告地址：`https://github.com/SproutSeeds/dormant-behavior-audit/releases/download/v1.0.0/dormant-behavior-audit-v1.0.0-reference-report.pdf` - 基准测试主页地址：`https://sproutseeds.github.io/dormant-behavior-audit/` - 发布状态：公开